Experiment Tracking & VersioningOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Experiment Tracking & Versioning

A training run is a black box that consumed code, data, and a stack of GPUs for eight hours and emitted a checkpoint. Six weeks later your boss asks "can we get this number back" and you discover the LR was in a notebook cell that no longer exists, the dataset was a CSV someone overwrote, and the CUDA driver on the box has been bumped. Experiment tracking and versioning are the disciplines that turn that black box into a reproducible recipe: every run is a row in a database with its hyperparameters, its metrics over time, its output artifacts, and (critically) content-addressed pointers back to the exact code, exact data, and exact environment that produced it.

Real-world examples: MLflow runs the experiment tracker at thousands of teams from startups to FAANGs; Weights & Biases is the default for hyperparameter sweeps in academic and applied DL labs; DVC versions datasets and model checkpoints in git-style content-addressed storage; Sacred is the quiet workhorse of research labs from Berkeley to ETH; Snowflake + dbt version SQL pipelines as code (the data-engineering equivalent of git); the Hugging Face Hub is the model-artifact registry the open-source world has converged on. Each tool solves a slice of the same problem — "what produced this model, and can I reproduce it?" — and a production team usually wires together two or three.

What an experiment tracker stores

An experiment tracker is not a magical thing. At its core it is a database with a small, standard schema:

  • Run. One execution of training code. Identified by a generated run id (UUID), the git SHA of the training repo at run time, and a timestamp. Everything else hangs off the run id.
  • Parameters / hyperparameters. The inputs to the run: model architecture, learning rate, batch size, optimizer choice, seed, dataset name, number of epochs. Logged once at the start. A change in any parameter is a different run.
  • Metrics. Scalar time-series logged during the run: training loss at each step, validation accuracy each epoch, gradient norms, GPU utilisation. Plus per-run summary metrics: final test AUC, best validation F1, total training time. Trackers store these as (metric_name, step, value) triples so they can be plotted as curves.
  • Artifacts. Files produced by the run: model weights, optimiser state, configs, generated plots, test reports, a CSV of failure cases, sample predictions. The artifact store is content-addressed by hash (or by run id + filename) so two runs that emit byte-identical weights de-duplicate to one blob on disk.
  • Tags / metadata. Human-readable annotations: experiment name ("baseline-v3"), owner, dataset version, environment fingerprint, free-form notes. These are how a human navigates a thousand runs six months later.

The point is not that the schema is complex (it isn't) but that all five of these dimensions have to be captured automatically. The moment any of them is "in the notebook" or "on the box that ran it," the run stops being reproducible.

The three things you must version to reproduce a run

Reproducibility means: a different person on a different machine at a different time can re-run your training code and get the same model (to the precision the hardware allows). Achieving that requires versioning three things, and forgetting any one of them is fatal.

  1. Code. The git SHA of the training repo at the moment training started, plus the SHA of any config file that lives outside the repo. "I ran the training script" without a SHA tells you nothing — the script changes daily.
  2. Data. A content-addressed pointer to the exact training set, validation set, and test set. Not "the data in s3://mybucket/training/" — that's a moving target. The right artifact is a hash, a DVC pointer, a LakeFS commit id, a Delta or Apache Iceberg table version, or an S3 object version id. Any of these can be rolled back to bit-identical bytes months later.
  3. Environment. A pinned dependency lock — requirements.txt with hashes, poetry.lock, a container image digest (sha256:...), the CUDA toolkit version, the GPU driver version. "Python 3.11 and PyTorch 2.4" is not enough; a minor torch patch can change which kernel is selected and silently move the last decimal of your metrics.
Versioning axisWhat it capturesTypical tool
CodeTraining script + configgit (SHA)
DataTraining/val/test snapshotDVC, LakeFS, Delta/Iceberg, S3 object versioning
EnvironmentLibraries, drivers, CUDAcontainer digest, pip-tools lockfile, conda lock

Forget any single one and your run is not reproducible. The common failure modes — "we know the SHA but the data changed," "we know the data and the SHA but our base image was rebuilt," "we know everything except the CUDA driver" — are how teams discover this lesson the hard way.

The three things to version

Code, data, environment. A reproducible run logs a git SHA, a content-addressed data pointer, and a pinned environment fingerprint (container digest or lockfile hash). If any of those three is missing from your tracker's run record, the run is not reproducible — full stop.

Reproducibility in practice

Even with all three axes pinned, true bit-for-bit reproducibility is harder than it sounds.

Seed everything

Set every source of randomness at the top of the script:

  • random.seed(s) for Python's stdlib.
  • np.random.seed(s) for NumPy (or, better, rng = np.random.default_rng(s) for newer code).
  • torch.manual_seed(s), torch.cuda.manual_seed_all(s) for PyTorch on CPU and all GPUs.
  • Framework-specific deterministic flags: torch.use_deterministic_algorithms(True), cudnn.deterministic = True, cudnn.benchmark = False.

Even so, GPU kernels routinely add non-determinism. Float summation is non-associative ((a + b) + c ≠ a + (b + c) in IEEE-754) and parallel reductions on a GPU choose summation order based on block scheduling, which depends on hardware and load. The last decimal of your loss can shift across two "identical" runs and there is nothing you can do about it short of forcing every reduction to a deterministic kernel (slow).

Pin software versions, not just package versions

torch==2.4.0 is a package version. torch==2.4.0+cu121 is closer. The container digest sha256:abc123... is what actually pins the software. "Pinning Python packages" without pinning the system layer underneath them is how teams ship "reproducible" pipelines that produce different metrics on the new cluster.

LLM training adds layers

LLM training pipelines add their own reproducibility traps. Distributed training depends on the order workers initialise (NCCL ring topology), gradient accumulation introduces boundary effects (where you split a batch matters), and dataloader sharding decides which examples each rank sees in which order. A run that is reproducible on a single GPU is not necessarily reproducible on 8, and reproducibility at 8 GPUs does not extend to 64. The fix is to version the distributed config (world size, gradient accumulation steps, micro-batch size, dataloader seed per rank) alongside the model config.

The hyperparameter sweep lifecycle

Most training runs are not standalone. They are members of a sweep: try ten learning rates, three seeds each, plot validation loss, pick the best. The tracker is the substrate that makes this manageable.

Search strategies

  • Grid search. Enumerate every combination of a discrete set of values. Embarrassingly parallel, easy to reason about, scales horribly: 5 LRs × 4 batch sizes × 3 seeds × 4 hidden sizes = 240 runs.
  • Random search. Sample each hyperparameter from a distribution. Surprisingly competitive with grid because most hyperparameters do not matter equally — random search spends its budget covering the dimensions that matter.
  • Bayesian / optimised search. Fit a surrogate model (Gaussian process, tree-structured Parzen estimator) on the runs you have, propose the next hyperparameter point with highest expected improvement, repeat. Best in tight-budget regimes.

Log every run, not just the winner

A common mistake is to only save the run that "won." This destroys the data that would let you tune the next sweep efficiently. The runs that failed are training data for your intuition: they show you where the LR was too high (loss diverged at step 200), where the model was capacity-starved (validation loss plateaued early), where the seed mattered (one seed crashed, three converged). All of that is useful, all of it lives in the tracker.

Early stopping vs full-budget runs

Two strategies for sweeps. Early-stop: kill any run whose validation loss is worse than the best run at the same step (median-rule, Hyperband, ASHA). Cheap, biased toward fast-improving runs. Full-budget: train every run to completion. Expensive, unbiased. Most teams start with early stopping for screening, then re-run the top-3 to completion to pick the final winner.

Artifact stores and model registries

A tracker records what happened during runs. A model registry records which model is allowed to serve traffic. These are different responsibilities, and confusing them is a frequent source of production incidents.

The promotion boundary

A run produced a model. The model registry decides whether that model is promoted to production. Crossing that boundary is a deliberate, governed act: a human (or an automated approval pipeline) signs off, and the registry records the promotion as an immutable event with the originating run id.

Stages

The canonical promotion ladder: staging → production → archived. A model in staging is being shadow-tested or canary-tested. A model in production is what serving infra loads when it boots. An archived model is kept for lineage but no longer serves. A run promoted to production should be content-addressed (its weights have a hash) and immutable: nobody is allowed to push new weights under the same model version.

Lineage

Every prediction made in production should be traceable backwards: prediction → model version → originating run → training data snapshot → upstream data pipeline. When something goes wrong (a regression, a fairness incident, a compliance audit), the lineage is the single most valuable piece of infrastructure you have. The runs and the registry are the substrate; the lineage is the graph you walk through them.

Promotion is the line between research and production

A run is research output. A registered, promoted model is a production contract. The instant you blur the line — pointing serving infra at the latest run's checkpoint, or letting any run author push a model to production — you have lost the audit trail that makes incident response possible. Treat the registry's promote operation like a git tag on a release: deliberate, auditable, immutable.

Exercise

Your tracker stores every training run with its hyperparameters and metric curves. Six weeks after a model is in production, an analyst asks 'which run produced *this* serving model, and on which data was it trained?' What is the missing layer your team has not yet implemented?

Need a hint?

A run is a research event. What turns a particular run's checkpoint into the *one* model that serving infra loads?

Exercise

Code, data, and environment are the three axes you must version to reproduce a training run. Suppose you have a $p = 0.9$ chance of correctly pinning each axis independently. Assuming the axes are independent, what is the probability of producing a fully reproducible run? Enter a decimal in [0, 1] rounded to two decimal places.

0probability0.50probability1probability
Need a hint?

All three independent events must succeed: probability is $p \cdot p \cdot p$.

Exercise

Your team's tracker stores every run's parameters and the path to its training table: `s3://mybucket/training/v3/`. Two months later, you re-run the same training script with the same git SHA and the same path — and get a model with measurably different test metrics. The team is confused: 'we pinned everything.' What is the most likely root cause?

Need a hint?

Is `s3://mybucket/training/v3/` actually a *version*, or is it just a path that can be overwritten?

Common mistakes

  1. "Magic" hyperparameters in checkpoint filenames. model_v3_final_FINAL_lr3e4_actually_use_this.pt. The LR is in the filename. There is no other record of it. The fix is to log every hyperparameter as a tracked parameter on the run, never inline it into a filename. Treat filenames as opaque keys.
  2. Re-running notebook cells out of order. Cell 12 saved the model. Cell 7 was re-run with different hyperparameters after. The saved model no longer matches the saved metrics, and you do not know which is correct. The fix is to run training from a script (python train.py --lr 3e-4 ...), not interactively; the run record is the script + its arguments.
  3. Logging the model file but not the preprocessing code. The model expects features normalised with the training mean and std. Six months later, only the model file remains; the preprocessing code is gone. At inference time the new code normalises differently and predictions are silently shifted. The fix is to log the preprocessor as an artifact alongside the model — or, better, bake it into the model graph so it cannot be lost.
  4. "Works on my machine." Pinned package versions but not pinned CUDA / driver / OS. The container digest is the right level of pinning; anything looser is theatre.
  5. No data version. "We trained on the production click logs from last Tuesday." Tuesday is not a version — tomorrow's Tuesday will have backfills, late-arriving events, schema migrations, deletions. The fix is a content-addressed snapshot the moment training reads the data: DVC commit, LakeFS branch, S3 object versions, Iceberg table version. A pointer to "the data" without a version is not a pointer; it is wishful thinking.
  6. Only logging the winning run. Throwing away failed runs is throwing away the data that would have made your next sweep cheaper. Log every run, even the ones that diverged at step 200.

Key takeaways

  • An experiment tracker is a database of runs. Five dimensions: run id, parameters, metrics over time, artifacts, tags. Every dimension must be captured automatically at training time — anything kept in a notebook or in someone's head is not captured.
  • Three things must be versioned to reproduce a run. Code (git SHA), data (content-addressed pointer), environment (container digest or pinned lock). Forget any one and the run is irreproducible.
  • Even with everything pinned, bit-for-bit reproducibility is hard. GPU non-determinism and float-summation order leak through. Seed everything, pin the system layer, version the distributed config; accept that the last decimal of the loss may still drift.
  • Sweeps are first-class. Log every run, not just the winner. The failed runs are training data for your next sweep.
  • The model registry is a separate layer from the tracker. A run is a research artifact; a promoted, registered model is a production contract. Cross the boundary deliberately.
  • Lineage is the single most valuable property to maintain. Prediction → model version → run → data snapshot. Lose it and your incident response is reduced to guessing.

Related concepts

  • ML Systems Design & Problem Framing: the systems lens that asks "what happens when something goes wrong." Lineage from prediction → model → run → data is exactly the property a system-design review will demand and that tracking + versioning makes possible.
  • Data Engineering Fundamentals: the layer that produces the snapshots experiment tracking points to. Content-addressed data versioning (Delta / Iceberg, DVC, LakeFS) sits on top of the storage decisions made there.
  • Training Data: Sampling, Labelling, Imbalance: every sampling and labelling decision is a transformation of the data version. The training set hash should be a function of both the source snapshot and the sampling/labelling code.
  • Deployment Pitfalls: train-serve skew, drift, and silent failures are exactly the pathologies a lineage trail lets you diagnose. Without it, all you can say is "the model got worse."

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
Experiment Tracking & VersioningLinear & Logistic Regress…Neural NetworksModel Evaluation & Valida…ML Systems Design & Probl…Data Engineering Fundamen…Training Data: Sampling, …Deployment PitfallsMLOps Infrastructure & Or…CI/CD, Continuous Trainin…