CI/CD, Continuous Training & MLOps MaturityOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

CI/CD, Continuous Training & MLOps Maturity

Classic software has two automated loops: CI (test and build on every commit) and CD (deploy the build). ML adds a third (CT, Continuous Training), because an ML system can decay even when its code never changes: the world shifts, the data drifts, and yesterday's model is wrong today. MLOps maturity is largely a story of automating these three loops.

Real-world examples: a fraud model retrained nightly on the latest labelled transactions; a recommender that auto-retrains when feature drift crosses a threshold; a forecasting pipeline that runs the full train-evaluate-deploy sequence from a single merged pull request.

The crucial shift: ship the pipeline, not the model

The defining idea of MLOps maturity is what artifact you deploy. In an immature setup you deploy a trained model — a person trains it in a notebook and hands the weights to ops. In a mature setup you deploy a training pipeline — an automated, reproducible sequence (ingest → validate → train → evaluate → register → serve) that produces models on demand.

Once the pipeline is the deployable unit, retraining on fresh data is a pipeline run, not a research project.

Google's MLOps maturity levels

Google's widely used framework defines three levels by how much of the ML process is automated:

LevelNameWhat's automatedRetraining cadence
0Manual processNothing: notebook-driven, manual hand-off to opsRare, manual
1ML pipeline automationThe training pipeline runs automatically with data/model validation; CT enabledFrequent, on a trigger
2CI/CD pipeline automationThe pipeline itself is built, tested, and deployed via CI/CDRapid, reliable, multi-pipeline
  • Level 0 is the most common starting point: a script or notebook produces a model, someone deploys it, and nobody retrains until performance visibly degrades. The training–serving skew and drift pitfalls from earlier lessons live here.
  • Level 1 automates the training pipeline. The pipeline runs on a trigger, includes data validation and model validation steps, and pushes a validated model to a registry. The team can now retrain frequently and reliably.
  • Level 2 automates building and deploying the pipeline code itself. A change to feature logic or model architecture flows through CI (unit/integration tests on pipeline components) and CD (deploy the new pipeline), so the system can evolve many pipelines rapidly.

Maturity is a means, not a goal

Level 2 is not automatically "better." A model retrained quarterly on stable data may be perfectly served at Level 0. Climb the levels when retraining frequency and change velocity justify the automation cost — the right level is the lowest one that keeps the model fresh enough.

What CI, CD, and CT actually test

Each loop gains ML-specific responsibilities beyond standard software practice:

  • CI: besides unit tests, validate pipeline components (does feature code produce the expected schema?), test that the model trains for a few steps, and check that converted/serialised models load.
  • CD: deploy the pipeline (Level 2) and the model service; verify the served model's prediction matches the offline model (no train–serve skew) before flipping traffic.
  • CT: run the training pipeline on a trigger and gate the result on validation.

A retraining trigger is one of:

retrain when  ∈  {schedule (e.g. nightly),  new data volume,  drift>τ,  performance<θ,  on demand}\text{retrain when} \;\in\; \{\text{schedule (e.g. nightly)},\; \text{new data volume},\; \text{drift} > \tau,\; \text{performance} < \theta,\; \text{on demand}\}

Automated retraining needs automated gates

CT without model validation is a loop that can silently push a worse model to production. Every retraining run must clear an evaluation gate — better than the current production model on a held-out set and above an absolute floor — before it is registered for deployment. Automate the promotion decision, don't automate around it.

Each of those pieces sounds obviously worth having, which is exactly why they are easy to skip. Build the loop below one stage at a time on 180 simulated days of a model in production — deploy it and watch it rot, add monitoring and find out how late it tells you, add a trigger and find out what that trigger cannot see, then add the two validation gates against two faults that break in completely different ways. The world control is the one that argues with the maturity ladder: on a steady world every trigger lands on the same accuracy, and all the automation buys nothing.

The continuous-training loop, stage by stage

Level 0 · a model in productionstep 1 of 6
A deployed model decays even though its code never changes

One model, trained once on the first three weeks and left alone — the Level 0 setup, where the artifact you ship is a trained model and a person is the retraining loop. Accuracy is measured every day on fresh traffic it has never seen.

Nothing about the model changes. What changes is the world, and the scenario control decides how much. That is the whole argument for adding a third loop to CI and CD: software rots when you edit it, models rot when you do not.

Switch the scenario control between Steady and Shock — same model, same code, opposite outcomes.

worldretrain trigger
Deploy a modelDrifting world · no retraining
0.20.40.60.8SLO 0.84day 21day 50day 80day 110day 140day 170accuracy
in production

Mean accuracy 0.702, worst day 0.376, 81 of 159 days below the SLO.

true accuracy today

180 simulated days of a model in production: a logistic classifier retrained on a rolling 21-day window, scored daily on fresh traffic, with labels arriving a week late. Two faults are planted in the training stream — a feature that collapses to a constant, and a stretch of noisy labels — because they fail differently and so justify different gates. Accuracy, PSI, every promotion decision and the scoreboard are computed from the simulation; the world and trigger controls re-run all of it.

Exercise

In Google's MLOps maturity framework, what defines Level 1 (vs Level 0)?

Need a hint?

Level 1 is about automating the training *pipeline*, with validation and continuous training.

Exercise

Which of these is a legitimate trigger for Continuous Training (CT)?

Need a hint?

CT exists because models decay from data/world changes.

Common mistakes

  1. Deploying models instead of pipelines: keeping a human in the training loop forever, so retraining never scales.
  2. CT without validation gates: automating retraining but not the check that the new model is actually better.
  3. Chasing Level 2 prematurely: paying for CI/CD-of-pipelines automation a stable, slow-moving model doesn't need.
  4. Skipping the skew check in CD: promoting a model whose served predictions differ from its offline evaluation.

Key takeaways

  • ML adds CT (Continuous Training) to CI/CD because models decay from data shift, not just code change.
  • The maturity jump is deploying a training pipeline, not a trained model.
  • Google's levels 0/1/2 track automation: manual → automated pipeline (CT) → automated CI/CD of pipelines.
  • Automated retraining is only safe with automated data and model validation gates.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelated
CI/CD, Continuous Training & MLOp…Linear & Logistic Regress…Neural NetworksModel Evaluation & Valida…Experiment Tracking & Ver…MLOps Infrastructure & Or…Deployment PitfallsModel Registry, Versionin…