Model Drift Explained (2026): Data vs Concept Drift, PSI, KS and LLMs

Every model I have seen fail quietly in production failed the same way: nothing crashed, latency was fine, the dashboards were green, and the predictions were slowly getting worse because the world had moved and the model had not. That is model drift, and in 2026 it has two very different faces. Classic ML models drift because their input data or the relationship they learned changes. LLM applications drift for those reasons too, plus a new one: the model behind your API call gets retired and replaced on someone else's schedule. This post explains data drift vs concept drift vs prediction drift, what the common statistical tests (PSI, KS, Wasserstein) actually measure and where they lie to you, how to think about LLM quality drift, which monitoring tools are still standing after a year of shutdowns and acquisitions, and how to decide when drift should trigger retraining.

What is model drift?

Model drift is the umbrella term for a model's real-world performance degrading after deployment because the data it sees no longer matches the data it was trained on. The model itself has not changed. Its weights are identical to the day you shipped it. What changed is the environment.

The hard part is that you usually cannot measure performance in real time. A fraud model learns the truth weeks later, when the chargeback arrives. So drift monitoring is mostly about proxies: watching inputs and outputs for signs the ground has shifted, before labels arrive to prove it. Each kind of drift needs different evidence.

Data drift vs concept drift vs prediction drift

In probability terms, a supervised model learns P(y | X), the chance of an outcome given the features. Drift is a change in one of the pieces of that picture.

Data drift (covariate shift, feature drift) is a change in P(X), the distribution of inputs. Your app launches in a new country and the age distribution shifts. The model may still be right about the relationship it learned, but it is being asked about regions of input space it rarely saw.

Concept drift is a change in P(y | X), the relationship itself. Fraudsters adapt, so a pattern that used to be safe becomes risky. Concept drift is the dangerous one, because the inputs can look perfectly stable while the model becomes wrong, and you generally cannot detect it without labels.

Prediction drift is a change in the distribution of the model's outputs, P(ŷ). If your approval model used to approve 30% of applications and now approves 45%, something changed, either in the inputs or in a pipeline bug. Prediction drift is cheap to measure and often the earliest useful alarm.

Two relatives get lumped in. Training-serving skew is production features computed differently from training features (a timezone bug, a missing join), so the model drifts on day one; that is a bug, not drift. Label shift is a change in the outcome's base rate, which may only need recalibration.

How do you detect data drift? PSI, KS, Wasserstein and friends

Every drift test compares a reference window (validation data or a known-good production period) against a current window, column by column, and produces a score. They differ in what the score means.

• Kolmogorov-Smirnov (KS) is a hypothesis test for numerical columns. It returns a p-value: the probability of seeing a difference this large if the two samples came from the same distribution. Below 0.05 means "drift".
• Population Stability Index (PSI) bins both samples and sums (current share minus reference share) times the log of their ratio. It comes from credit scoring, where 0.1 and 0.25 are the traditional "watch" and "act" bands.
• Wasserstein distance (earth mover's distance) measures how far you would have to move probability mass to turn one distribution into the other. Normalised by the reference standard deviation, 0.1 means "the distribution moved about a tenth of a standard deviation".
• Jensen-Shannon distance and chi-squared do the equivalent job for categorical columns.

The single most useful fact about these tests is that p-value tests get more trigger-happy as data grows, while distance metrics do not. With enough rows, KS will flag a shift too small to matter to any model. I ran this myself to show it, a 0.05 standard deviation nudge to a normally distributed feature (numpy 2.5.4 and scipy 1.18.1, the versions pip resolved today):

# drift_by_hand.py: PSI, KS and Wasserstein on the same tiny shift
import numpy as np                      # numpy==2.5.4
from scipy.stats import ks_2samp, wasserstein_distance   # scipy==1.18.1

def psi(ref, cur, bins=10):
    # bin edges from reference quantiles, so each reference bin holds ~10% of rows
    edges = np.quantile(ref, np.linspace(0, 1, bins + 1))
    edges[0], edges[-1] = -np.inf, np.inf
    r = np.histogram(ref, edges)[0] / len(ref)
    c = np.histogram(cur, edges)[0] / len(cur)
    r, c = np.clip(r, 1e-6, None), np.clip(c, 1e-6, None)   # avoid log(0)
    return float(np.sum((c - r) * np.log(c / r)))

rng = np.random.default_rng(0)
for n in (500, 50_000):
    ref = rng.normal(40, 10, n)          # e.g. customer age at training time
    cur = rng.normal(40.5, 10, n)        # a 0.05 standard-deviation nudge
    print(f"n={n:>6}  PSI={psi(ref, cur):.4f}  "
          f"KS p={ks_2samp(ref, cur).pvalue:.4f}  "
          f"W/std={wasserstein_distance(ref, cur) / ref.std():.4f}")

# Output on my machine:
# n=   500  PSI=0.0285  KS p=0.6659  W/std=0.0746
# n= 50000  PSI=0.0025  KS p=0.0000  W/std=0.0483

Same shift, a hundred times more rows, and KS flips to "drift detected" while PSI and Wasserstein stay under 0.1. That matches what Evidently found in a larger experiment on real datasets: KS raised alerts for a 0.5% change once a dataset passed 100,000 objects, PSI ignored small changes at any size, and Wasserstein sat in the middle. It is also why Evidently's default algorithm switches methods by sample size: KS and chi-squared for references of 1,000 rows or fewer, normed Wasserstein and Jensen-Shannon with a 0.1 threshold above that.

Method	Column type	Score	Default drift rule	Behaviour at large n	Source
KS test	Numerical	p-value	p below 0.05	Flags trivial shifts	Evidently drift methods
Chi-squared	Categorical	p-value	p below 0.05	Flags trivial shifts	Evidently drift methods
PSI	Both	Index	0.1 in Evidently; 0.1 and 0.25 by convention	Stable, low sensitivity	Yurdakul and Naranjo
Wasserstein (normed)	Numerical	Distance in std units	0.1 or more	Stable above ~100k rows	Evidently experiment
Jensen-Shannon	Both	Distance, 0 to 1	0.1 or more	Stable	Evidently drift methods
Domain classifier	Text	ROC AUC	Above 0.55 (over 1,000 rows)	Stable	Evidently drift methods, after Rabanser et al.
JS divergence / L-infinity	Num / Cat	Distance	0.3 (Vertex v1 default)	Stable	Vertex AI Model Monitoring

Be sceptical of the PSI bands everyone quotes. The 0.1 and 0.25 cut-offs trace back to 1990s credit scorecard practice, and Yurdakul and Naranjo's paper on the statistical properties of PSI notes they ignore error rates, though the statistic depends on bin count and sample size. Treat any threshold as a starting point tuned against your own history.

Does data drift mean the model is broken?

No, and this is where most drift monitoring goes wrong. A feature can drift a lot and not matter because the model barely uses it, or drift a little and matter enormously because it is the top split. So I treat drift signals as a funnel, not a verdict:

1. Data quality first. Nulls, type changes and schema breaks cause more "drift" alerts than the world does, and most drift tests drop nulls before comparing. Run explicit missing-value checks.
2. Prediction drift second. If the outputs have not moved, input drift is often not urgent.
3. Weight by importance. Alert on the handful of features that drive the model, and report the rest.
4. Confirm with quality when labels arrive. Drift is a hypothesis. Labelled accuracy, F1 or MAE is the proof. Tools like NannyML's performance estimation try to estimate accuracy before labels arrive using the model's own confidence, which helps for classifiers whose probabilities are well calibrated.

I would rather have three drift alerts that page someone than forty that everyone mutes. The principles from observability vs monitoring apply directly: monitor symptoms that matter to users, keep the diagnostic detail one click away.

How do LLMs drift?

LLM applications drift in all the classic ways. Your users' questions change, your retrieval corpus changes, your prompt changes. But there is a fourth source that classic ML rarely had: the model changes underneath you.

The best-known evidence is still the 2023 study by Chen, Zaharia and Zou, How Is ChatGPT's Behavior Changing over Time?. Comparing the March and June 2023 versions of GPT-4, they found accuracy on identifying prime numbers fell from 84% to 51%, with more formatting mistakes in generated code. Same product name, different behaviour, three months apart.

Providers have responded by pinning. Anthropic's model IDs page states that each Claude model ID identifies a pinned version (the convenience aliases for older models excepted), and its models overview gives each model a "not sooner than" retirement date. That removes silent drift but guarantees scheduled drift: when a snapshot retires, you migrate, and the new model behaves differently. On 14 October 2026 OpenAI's deprecations page announced shutdown of GPT-4o and GPT-4.1 snapshots, including gpt-4o-2024-08-06 and gpt-4.1-2025-04-14, for 14 January 2027. If your features call those, you have a drift event on your calendar.

Statistical drift tests do not apply directly to free text, so LLM drift monitoring works on derived signals:

• Descriptors: per-response numbers such as length, refusal rate, language, presence of required fields or sentiment. These turn text into columns you can run ordinary drift tests on.
• Eval scores on a fixed test set, run before and after every model, prompt or retrieval change. This is the regression-test half of the problem, covered in MLOps in 2026: evals are the new unit tests.
• LLM-as-judge scores on a sample of production traffic, with human spot checks, because judges drift too.
• Domain classifier drift on text: train a classifier to tell last month's prompts from this month's; if it succeeds (ROC AUC well above 0.5), your traffic changed. Evidently uses this Failing Loudly method for text columns.

The DevOps parallel is exact: a model migration is a dependency upgrade with no behavioural changelog. Do not bump a model ID without running your evals.

Model drift monitoring tools in 2026: what is still standing

The tool market thinned out in the last eighteen months, which matters if you are choosing something to run for years.

Tool	Status (October 2026)	License	Best at	Source
Evidently	v0.7.23 (11 Sep 2026); hosted Cloud SaaS no longer offered, self-host the UI	Apache-2.0	Drift reports, tests, LLM descriptors	GitHub release, docs
NannyML	v0.13.1 (12 Jul 2025); company acquired by Soda on 9 Jun 2025	Apache-2.0	Performance estimation without labels	PyPI, Soda
WhyLabs	Company "discontinuing operations"; platform open-sourced	Apache-2.0	whylogs profiles (last release Dec 2024)	WhyLabs, whylabs-oss
Arize Phoenix	v21.0.0 (30 Oct 2026)	Elastic-2.0	LLM tracing and evals	PyPI
SageMaker Model Monitor	Closed to new customers; AWS points to Evidently-based samples	Managed	Existing AWS users only	AWS docs
Vertex AI Model Monitoring	v1 GA, v2 in preview	Managed	GCP-hosted models	Google Cloud docs

Two details stood out. AWS's replacement guidance for SageMaker Model Monitor is built on MLflow plus Evidently's DataDriftPreset, scheduled by EventBridge and Lambda; when the hyperscaler's migration path is "run the open-source library yourself", the category has settled. And the libraries are not frozen in amber. Evidently 0.7.23 fails to import with pandas 3.1.0, released on 22 October 2026, an open issue at the time of writing. Pin your dependencies, as you would for anything else in production.

For LLM tracing specifically, there are more options than this table covers. I compare them in LLM observability tools compared.

When should drift trigger retraining?

Retraining on every drift alert is the ML equivalent of restarting the server on every warning: it hides the cause and can make things worse if the new window contains a data bug. A sensible policy has three tiers.

Scheduled retraining covers slow, expected drift: retrain on a calendar that matches your domain, with the same eval gate every time. Boring, and it handles most cases.

Triggered retraining covers sudden drift that matters. A reasonable trigger combines signals rather than relying on one: the share of drifted important features crosses a threshold, prediction drift confirms it, and, where labels exist, quality on the latest labelled window drops below an agreed floor. AWS's sample pipelines, for instance, alert when more than 30% of features drift, with separate floors on F1 and accuracy. Those numbers are a sample's defaults, not advice; the point is the shape of the rule.

Investigation, not retraining is the right response for schema changes, pipeline bugs and training-serving skew. Retraining on broken data teaches the model the bug.

Whatever triggers it, retraining needs normal release discipline: versioned data, a model registry (see my MLflow 3 guide), an eval gate, a canary period and a rollback. Drift monitoring tells you when to look, not that the new model is better.

FAQ

What is the difference between data drift and concept drift?

Data drift is a change in the distribution of a model's inputs, P(X). Concept drift is a change in the relationship between inputs and the target, P(y given X). Data drift can be detected without labels using statistical tests on features; concept drift usually needs labels, because the inputs can look unchanged while the model becomes wrong.

What is a good PSI threshold for drift?

By convention, PSI below 0.1 means stable, 0.1 to 0.25 means a moderate shift worth watching, and above 0.25 means a significant shift. These bands come from credit scoring practice and have no formal statistical basis, so tune them against your own history and sample sizes.

Why does my drift monitor alert on every batch?

The most common cause is a p-value test such as KS or chi-squared on large samples, which flags differences far too small to affect a model. Switch to a distance metric such as Wasserstein or Jensen-Shannon, sample the data down, or alert only on the features your model depends on.

Do LLMs suffer from model drift?

Yes. LLM applications drift when traffic, prompts or retrieved data change, and when a pinned provider snapshot retires and you migrate. Monitor descriptors such as length and refusal rate, and run a fixed eval set before every model change.

This quarter, if I owned a production model, I would do three things. Add a prediction drift check and a missing-value check, because they are cheap and catch most real problems. Replace any KS-on-everything drift job with distance metrics on the top features. And for every LLM feature, write down which model snapshot it calls and when that snapshot retires, then put the migration and its eval run on the calendar now rather than in the first week of January. The hands-on version, with Evidently code you can run, is in my companion guide: how to monitor data drift with Evidently.

Related: MLOps in 2026: Evals Are the New Unit Tests, MLOps vs LLMOps vs AgentOps in 2026 and How to Use MLflow 3: Tracking, Model Registry and GenAI Tracing.