LoginSign Up
← Back to all sparks
C

ClearML

AI-ASSISTANTS
Velocity5.0

End-to-end MLOps platform for experiment tracking and orchestration

clear.ml ↗

ClearML is filling in the hyperdataset lifecycle while hardening the SDK against what it loads.

experiment trackinghyperdatasetsartifact securitystorage managersdk maintenance
◆Current state
Recent releases pair hyperdataset work with a steady security pass over the SDK's own inputs. 2.1.7 added an opt-out that blocks processing of pickled artifacts via call argument, config key or CLEARML_BLOCK_PICKLED_ARTIFACTS, plus a path-traversal check when import_offline_session extracts a zip; 2.1.6 added integrity-hash verification for pickled DataFrame artifacts; 2.1.8 added a path-traversal check in dataset merging. The hyperdataset API meanwhile keeps accumulating lifecycle operations — tagging, version snapshots, single-call publishing, DataView retrieval, and now entry deletion, metadata get/set, mapping-rule management and an iterator.
◆Where it's heading
Two things are converging. The hyperdataset API is filling in the operations a dataset abstraction needs before anyone builds on it seriously: create, snapshot, tag, publish, retrieve, iterate, delete. That the newest release is mostly deletion and metadata management says the API is past the demo stage and into the parts people hit in production. Meanwhile the SDK is being treated as something that consumes untrusted input, because in a shared experiment tracker it is: an artifact is a file another user uploaded, and Python's default answer to a pickle is to execute it.
◆Prediction
Pickle blocking is opt-out today and the notes give no timeline for flipping the default. The clearer near-term threads are Python 2 removal and the f-string migration, both described as work in progress across several releases.

◆Recent moves

  1. 14h ago

    Hyperdataset entry deletion, metadata management and mapping rules

    The hyperdataset API gains the operations that were conspicuously missing: deleting data entries, getting and setting metadata, managing mapping rules, labelling dataviews and iterating a hyperdataset. Race-condition fixes in dataviews land alongside. This is the unglamorous half of an API becoming usable rather than demonstrable.

    View source ↗
  2. 12d ago

    In-memory streaming in the storage manager, DataView retrieval

    In-memory data streaming lands in the storage manager and DataView gets a retrieval method against the API server, with a 100 MB cap placed on registration request payloads. The cap is the detail worth noting — it implies people were pushing registration payloads large enough to need one.

    View source ↗
  3. 12d ago

    HPO trial pruning and hashlib usedforsecurity fixes

    Hyperparameter optimisation gains trial pruning for resilience, and hashlib calls are marked usedforsecurity=False throughout — a small change that matters on FIPS-restricted hosts. A parent_ids argument is deprecated in favour of parent_id without breaking existing callers.

    View source ↗
  4. 2mo ago

    Hyperdataset version snapshots and a static route validator

    Version snapshots give hyperdatasets a point-in-time handle, which is the prerequisite for reproducible references to a dataset that keeps changing. The f-string migration and dead-code removal continue in the background.

    View source ↗
  5. 2mo ago

    Hyperdataset tagging and publishing, plus Azure default credentials

    Tagging and single-call publishing extend the hyperdataset lifecycle, DefaultAzureCredential support removes a credential-plumbing step on Azure, and a path-traversal check lands in dataset merging — the same defensive thread running through the 2.1.6 and 2.1.7 releases.

    View source ↗
  6. 3mo ago

    Opt-out blocking for pickled artifacts and zip path traversal

    Users can now refuse to deserialise pickled artifacts, configurable per call, per config file or by environment variable, and offline-session zip extraction gets a path-traversal check. Making it opt-out rather than default keeps existing pipelines working while giving security-conscious deployments a switch to throw.

    View source ↗