Data drift in production: knowing when a model has aged

A broken backend service usually tells you: 500 errors, growing queues, alerts. A degraded machine learning model tells you nothing. It keeps answering in milliseconds, in the same format, just with steadily worse predictions. The most common cause is drift: production data stops resembling the data the model was trained on.

Three kinds of change

• Covariate drift: the distribution of inputs changes. A credit model trained before an interest-rate policy change starts seeing customers with different profiles.
• Label drift: the proportion of outcomes changes. Default rates rise even though customers look the same.
• Concept drift: the relationship between inputs and outcome changes. The same characteristics start to indicate different behavior, such as purchasing habits after a shift in sales channel.

Concept drift is the most dangerous and the hardest to detect, because it only shows up once the real outcome is known, often weeks later.

The delayed-label problem

In many business settings the outcome arrives late. You only learn whether a customer defaulted months after the prediction. Waiting for labels to measure performance means discovering the problem too late.

So monitoring combines two layers:

1. Immediate signals: the distributions of inputs and predictions, compared against a reference. They don't say the model is wrong, but they say it's operating outside known territory.
2. Delayed signals: performance metrics computed when labels arrive, per prediction-date cohort.

Measuring drift with PSI

The Population Stability Index compares a variable's distribution between a reference (for example, the validation period) and a production window. The variable is split into bins, and the index sums, for each bin, the difference in shares weighted by the log of their ratio.

import numpy as np
 
 
def psi(reference: np.ndarray, current: np.ndarray, bins: int = 10, eps: float = 1e-6) -> float:
    """Population Stability Index. Bin edges come from reference quantiles."""
    edges = np.unique(np.quantile(reference, np.linspace(0, 1, bins + 1)))
    edges[0], edges[-1] = -np.inf, np.inf
    ref_share = np.histogram(reference, edges)[0] / len(reference)
    cur_share = np.histogram(current, edges)[0] / len(current)
    ref_share = np.clip(ref_share, eps, None)
    cur_share = np.clip(cur_share, eps, None)
    return float(np.sum((cur_share - ref_share) * np.log(cur_share / ref_share)))

A convention widely used in credit risk treats PSI below 0.1 as stable, 0.1 to 0.25 as a moderate shift, and above 0.25 as a significant shift. These are rules of thumb, not statistical truths: calibrate thresholds on your own model's history before turning them into alerts.

For continuous variables, the Kolmogorov-Smirnov test is an alternative. At high volume, though, it flags statistically significant differences that don't matter in practice, so look at effect size, not only the p-value.

What to monitor

Monitoring every variable with equal weight creates noise. Prioritize:

• The predictions. A shift in the output distribution is the signal closest to business impact.
• The most important features for the model, according to the importance analysis done at training time.
• Data quality: share of missing values, categories never seen in training, out-of-range values. Many "drifts" are really a source system that changed a field's format.

Training-serving skew

Before suspecting the world, suspect the pipeline. Training-serving skew happens when a feature is computed one way in training and another way in production: a different time zone, a different aggregation window, a different default for missing values.

The best defense is computing features with the same code in both contexts and logging, in production, the inputs exactly as the model received them. Then you can compare them directly with the training data.

From alert to decision

A drift alert doesn't mean "retrain now". It means "investigate". The playbook we usually follow:

1. Check whether the change comes from a data problem (format, missing values, a broken integration).
2. If the change is real, estimate impact: has performance dropped in cohorts that already have labels?
3. Decide between retraining on recent data, adjusting the decision threshold or, for structural changes, revisiting the features.

Retraining automatically on every alert sounds convenient, but it can bake into the model a data problem that should have been fixed at the source. Prefer scheduled retraining, with the same evaluation used in development, and treat extraordinary retraining as a deliberate decision.

Summary

Models age silently. Monitoring input and prediction distributions against a reference, with metrics such as PSI, catches change before labels confirm the damage. Checking data quality and training-serving consistency first avoids false alarms, and treating an alert as the start of an investigation keeps retraining a decision rather than a reflex.