Concept Drift & AdaptationOpen in Colab
ML project loop
Data·Hypothesis space·Objective·Optimization·Evaluation·Deployment feedback

Concept Drift & Adaptation

Batch ML assumes the training and serving distributions match. On a live stream they routinely do not: user behaviour shifts, fraud patterns evolve, sensors age, a pandemic rewrites demand overnight. When the data-generating process changes over time, we call it concept drift, and a model that ignores it silently rots. This lesson covers detecting drift and adapting to it, the concerns at the heart of libraries like river.

The forecasting thread

This is the exact failure the site's running daily-demand model keeps hitting. The canonical series (from the project-loop hub) has a regime shift at day 400 — a permanent level jump. To a batch model it's invisible until the error blows up; a streaming system running a drift detector on the residuals catches it within days and adapts online. Concept drift is the demand problem's day-400 shift, generalized to "the world keeps changing."

What is drifting?

Write the joint distribution the stream is drawn from at time tt as Pt(X,Y)=Pt(X) Pt(Y∣X)P_t(X, Y) = P_t(X)\,P_t(Y \mid X). Drift is any change in PtP_t over time, and it comes in distinct flavours that call for different responses:

  • Virtual drift / covariate shift: P(X)P(X) changes but P(Y∣X)P(Y \mid X) does not. The input distribution moves (new users, new regions) yet the decision boundary is still correct. Often you can tolerate this.
  • Real / concept drift: P(Y∣X)P(Y \mid X) changes. The relationship between features and label is different now; the old decision boundary is wrong. This is the dangerous one.
  • Label / prior drift: P(Y)P(Y) changes (e.g. the base rate of fraud rises).

Drift also has a temporal shape:

  • Sudden: an abrupt switch (a policy change, a sensor swap).
  • Gradual: the new concept slowly displaces the old.
  • Incremental: a continuous slow morph.
  • Recurring: old concepts return (weekday vs weekend, seasonal).

Drift ≠ noise, and covariate shift ≠ concept drift

The hard part of detection is distinguishing a genuine change in the concept from ordinary sampling noise. And a spike in P(X)P(X) alone (covariate shift) may need no action, while a change in P(Y∣X)P(Y\mid X) demands retraining. Monitoring input statistics is not enough. You have to watch the model's error on labelled feedback.

Detecting drift

Detectors watch a signal (usually the model's streaming error rate) and raise an alarm when it changes more than chance allows.

DDM: Drift Detection Method

DDM tracks the online error rate ptp_t and its standard deviation st=pt(1−pt)/ts_t = \sqrt{p_t(1-p_t)/t}. For a stationary concept a well-fit classifier's error should not grow, so DDM records the minimum pmin⁡+smin⁡p_{\min} + s_{\min} seen so far and flags:

  • a warning when pt+st≥pmin⁡+2 smin⁡p_t + s_t \geq p_{\min} + 2\,s_{\min} (start buffering new data for a fresh model),
  • drift when pt+st≥pmin⁡+3 smin⁡p_t + s_t \geq p_{\min} + 3\,s_{\min} (replace the model).

The 2σ2\sigma/3σ3\sigma bands come straight from a normal approximation to the error rate.

ADWIN: Adaptive Windowing

ADWIN keeps a window WW of recent values and asks: is there a split of WW into an older part W0W_0 and a newer part W1W_1 whose means differ by more than chance? It drops the old part exactly when

∣μ^W0−μ^W1∣  >  ϵcut,ϵcut=12mln⁡4 ∣W∣δ,|\hat\mu_{W_0} - \hat\mu_{W_1}| \;>\; \epsilon_{\text{cut}}, \qquad \epsilon_{\text{cut}} = \sqrt{\frac{1}{2m}\ln\frac{4\,|W|}{\delta}},

where mm is the harmonic mean of the two sub-window sizes and δ\delta is a confidence parameter. When the test fires, ADWIN shrinks the window to just the recent segment, so the window length itself becomes an estimate of "how far back the current concept extends." A clever bucket scheme (exponential histograms, as in DGIM) keeps its memory logarithmic in the window size.

Concept drift: detect, then adapt

0.00.10.20.30.40.5stream position t →error rateconcept shiftsdrift detectedstatic (no adapt)adaptive (ADWIN)
true driftt = 120
detected att = 179
detection delay59

At the dashed line the concept changes. A static model (rose) never recovers — its error stays elevated. The adaptive model (teal) runs an ADWIN-style detector on its error stream; when two halves of its window diverge by more than the Hoeffding cut it flags drift (flag), drops the stale window, retrains, and error falls back to baseline. Raise the confidence δ to trip the detector sooner — at the cost of more false alarms on stationary data.

The confidence δ\delta is the classic tuning knob: raise it and ϵcut\epsilon_{\text{cut}} shrinks, so drift is caught sooner but false alarms on stationary data rise; lower it and the detector is conservative, adapting late but rarely crying wolf. This is the detection-delay vs false-alarm trade-off every drift detector must navigate.

Deep dive
ADWIN: Adaptive Windowing & the Hoeffding Cut
Includes runnable notebook

Adapting to drift

Detection is half the job; the model then has to change. Strategies, roughly in order of sophistication:

  1. Retrain on a window. Keep the last NN examples (or the post-ADWIN window) and refit. Simple, and the window automatically forgets stale data.
  2. Incremental / online update. Use an online learner (previous lesson) that continuously tracks the current concept; a decaying step size or explicit forgetting factor down-weights the past.
  3. Detect-and-reset. Pair any learner with a detector (ADWIN/DDM); on an alarm, reset or spin up a fresh model on recent data. This is what the animation above shows.
  4. Ensembles. Maintain a pool of models trained on different periods and weight them by recent accuracy (e.g. Dynamic Weighted Majority, Adaptive Random Forest). Recurring concepts are handled gracefully because an old expert regains weight when its concept returns.

Hoeffding trees: decision trees for streams

A batch decision tree needs all the data to choose each split. The Hoeffding tree (Very Fast Decision Tree) instead decides how many examples are enough. The Hoeffding bound says that after nn independent observations of a quantity in range RR, the true mean is within

ϵ=R2ln⁡(1/δ)2n\epsilon = \sqrt{\frac{R^2 \ln(1/\delta)}{2n}}

of the observed mean, with probability 1−δ1-\delta. So the tree accumulates split-quality statistics (e.g. information gain GG) at a leaf and splits as soon as the best attribute beats the runner-up by more than ϵ\epsilon: G(best)−G(2nd)>ϵG(\text{best}) - G(\text{2nd}) > \epsilon. That guarantees, with high probability, the same split the batch tree would have chosen — using only a small prefix of the stream and O(1)O(1) memory per example. The Hoeffding Adaptive Tree (HAT) adds an ADWIN detector at each node, growing an alternate subtree when drift is detected and swapping it in once it wins.

Python: adaptive learning with river

Python
from river import tree, drift, metrics

model    = tree.HoeffdingAdaptiveTreeClassifier()  # HT + per-node ADWIN
detector = drift.ADWIN(delta=0.002)
metric   = metrics.Accuracy()

for x, y in stream:                 # one pass over an evolving stream
    y_pred = model.predict_one(x)
    metric.update(y, y_pred)
    model.learn_one(x, y)           # incremental update

    detector.update(int(y_pred != y))   # feed the error signal
    if detector.drift_detected:
        print(f"drift detected — accuracy now {metric.get():.3f}")

Exercise

A recommender sees a wave of new users from a new country: the input distribution P(X) shifts noticeably, but the mapping from features to preferences P(Y|X) is unchanged. What kind of drift is this, and what does it imply?

Need a hint?

Which distribution actually changed — P(X) or P(Y|X)? Is the decision boundary still correct?

Exercise

ADWIN cuts its window when |μ̂_W0 − μ̂_W1| > ε_cut, where ε_cut = √( (1/2m)·ln(4|W|/δ) ). You increase the confidence parameter δ. What is the effect?

Need a hint?

A larger δ makes ε_cut smaller. A smaller threshold is easier to exceed.

Key takeaways

  • Concept drift is change in Pt(X,Y)P_t(X,Y). Distinguish covariate shift (P(X)P(X)) from dangerous real drift (P(Y∣X)P(Y\mid X)); note whether it is sudden, gradual, incremental, or recurring.
  • DDM flags drift when the error rate exceeds 2σ2\sigma/3σ3\sigma bands; ADWIN drops the stale window when two sub-window means differ by more than a Hoeffding cut ϵcut\epsilon_{\text{cut}}.
  • The detector's confidence δ\delta trades detection delay against false alarms.
  • Adapt by windowed retraining, online updates, detect-and-reset, or ensembles. Hoeffding trees grow decision trees on streams using the Hoeffding bound to decide when a split is justified.

Related concepts

Concept neighbourhood

How this concept connects — click any node to jump there, or explore the whole concept map.

Builds onRelatedGo deeper
Concept Drift & AdaptationOptimization for MLML in PracticeStreaming Algorithms & Sk…Online Learning & RegretStreaming ML in ProductionDecision TreesThe ML Project LoopADWIN: Adaptive Windowing…