How to Monitor Data Drift with Evidently 0.7: Step-by-Step (2026)

This guide is for ML and platform engineers who have a model in production and want drift monitoring they can run themselves, without a SaaS contract. By the end you will have a drift report comparing production data to a reference set, thresholds tuned so it does not cry wolf, a script that fails a CI job when drift crosses a line, a scheduled job writing snapshots to a workspace, a self-hosted dashboard, and a pattern for monitoring LLM outputs with descriptors. Everything below was run on 31 October 2026 with Evidently 0.7.23 (released 11 September 2026), Python 3.12, pandas 3.0.6 and scikit-learn 1.9.1, and the outputs shown are real. You need Python 3.10 or newer and a dataset you can load into pandas. If you want the theory first (data drift vs concept drift, what PSI and KS measure), read the companion explainer, model drift explained.

Step 1: Install Evidently and pin pandas

Evidently 0.7 (April 2025) made a new API the default, per the migration guide: column_mapping, TestSuite and evidently.metric_preset are gone, replaced by Report, Dataset, DataDefinition and evidently.presets. Many tutorials still use the old API, so check imports before you copy anything.

Pin both Evidently and pandas. pandas 3.1.0 shipped on 22 October 2026 and removed an internal function Evidently imports, so a fresh unpinned install fails on import evidently (issue #1974, with a fix still in review in PR #1978 at the time of writing).

python3.12 -m venv .venv && source .venv/bin/activate
pip install "evidently==0.7.23" "pandas==3.0.6" "scikit-learn==1.9.1" "pyarrow==26.0.0"
python -c "import evidently, pandas; print(evidently.__version__, pandas.__version__)"

Expected output:

0.7.23 3.0.6

If you skip the pandas pin, you get this:

ImportError: cannot import name 'infer_dtype_from_object' from 'pandas.core.dtypes.common'

Put the same pins in requirements.txt so CI and the scheduled job resolve identical versions. The rest of this guide assumes that file exists.

Step 2: Prepare reference and current datasets

Drift is always relative. You compare a current batch (yesterday's predictions log, this week's feature table) against a reference batch (validation data, or a known-good production window). Pick the reference deliberately: if it is training data that was heavily resampled or filtered, every production batch will look drifted.

Declare column types explicitly with DataDefinition. Evidently can infer them, but inference treats a numeric customer_id as a feature and a low-cardinality numeric code as numerical, both of which produce misleading drift scores.

# drift_check.py: compare a current batch with a reference batch
import numpy as np
import pandas as pd
from evidently import Dataset, DataDefinition, Report
from evidently.presets import DataDriftPreset

# Synthetic data so the example runs anywhere. Swap in pd.read_parquet(...)
rng = np.random.default_rng(42)
n = 5000
ref = pd.DataFrame({
    "age": rng.normal(40, 10, n).round(),
    "income": rng.lognormal(10, 0.5, n),
    "sessions": rng.poisson(3, n),
    "plan": rng.choice(["free", "pro", "team"], n, p=[0.6, 0.3, 0.1]),
    "country": rng.choice(["IN", "US", "DE", "BR"], n),
})
cur = ref.sample(n, replace=True, random_state=7).reset_index(drop=True)
cur["income"] = cur["income"] * 1.35                       # simulated shift
cur["plan"] = rng.choice(["free", "pro", "team"], n, p=[0.4, 0.4, 0.2])

schema = DataDefinition(
    numerical_columns=["age", "income", "sessions"],
    categorical_columns=["plan", "country"],
)
reference = Dataset.from_pandas(ref, data_definition=schema)
current = Dataset.from_pandas(cur, data_definition=schema)

Step 3: Run a Report with DataDriftPreset

Add the preset to a Report and run it. Argument order matters: report.run(current, reference), current first. Swapping them does not raise an error, it just answers a different question.

# drift_check.py (continued)
report = Report([DataDriftPreset()], include_tests=True)  # include_tests adds pass/fail
snapshot = report.run(current, reference)
snapshot.save_html("drift_report.html")    # open in a browser
d = snapshot.dict()                         # or snapshot.json()
for m in d["metrics"]:
    print(m["metric_name"], "=>", m["value"])
for t in d["tests"]:
    print(t["status"], "|", t["name"])

Output from my run:

DriftedColumnsCount(drift_share=0.5) => {'count': 2.0, 'share': 0.4}
ValueDrift(column=age,method=Wasserstein distance (normed),threshold=0.1) => 0.0122
ValueDrift(column=income,method=Wasserstein distance (normed),threshold=0.1) => 0.6419
ValueDrift(column=sessions,method=Wasserstein distance (normed),threshold=0.1) => 0.0253
ValueDrift(column=plan,method=Jensen-Shannon distance,threshold=0.1) => 0.1632
ValueDrift(column=country,method=Jensen-Shannon distance,threshold=0.1) => 0.0071
TestStatus.SUCCESS | Share of Drifted Columns: Less 0.500
TestStatus.SUCCESS | Value Drift for column age
TestStatus.FAIL | Value Drift for column income
TestStatus.SUCCESS | Value Drift for column sessions
TestStatus.FAIL | Value Drift for column plan
TestStatus.SUCCESS | Value Drift for column country

(I trimmed the long decimals.) Read it like this. Evidently picked normed Wasserstein for numerical columns and Jensen-Shannon for categorical ones, because the reference has more than 1,000 rows; with 1,000 or fewer it would use KS, chi-squared and a Z-test instead, per the drift explainer. Both shifted columns fail their column tests. The dataset-level test passes, because only 2 of 5 columns (40%) drifted and the default drift_share is 0.5. That default is often too lax for a small feature set, which is the next step.

Step 4: Tune drift methods and thresholds

Defaults are a starting point. The parameters that matter, from the customisation docs:

Parameter	What it does	Example
drift_share	Share of drifted columns that counts as dataset drift (default 0.5)	drift_share=0.3
method, threshold	One method and threshold for every column	method="psi", threshold=0.2
num_method, num_threshold	Override for numerical columns	num_method="wasserstein"
cat_method, cat_threshold	Override for categorical columns	cat_method="jensenshannon"
per_column_method, per_column_threshold	Dict of overrides for named columns	per_column_threshold={"income": 0.2}
columns	Only test these columns	columns=["income", "plan"]

from evidently import Report
from evidently.presets import DataDriftPreset
from evidently.metrics import MissingValueCount
from evidently.tests import lte

report = Report([
    DataDriftPreset(
        columns=["income", "sessions", "plan"],     # top features only
        num_method="wasserstein", num_threshold=0.1,
        cat_method="psi", cat_threshold=0.2,
        per_column_threshold={"income": 0.15},    # stricter on the key feature
        drift_share=0.3,
    ),
    # Drift tests drop nulls before comparing, so test nulls separately
    MissingValueCount(column="income", share_tests=[lte(0.01)]),
], include_tests=True)

Two rules I follow. Avoid p-value methods (ks, chisquare) on large batches, because they flag shifts too small to matter; I show the numbers for this in the explainer. And always pair drift with data-quality tests. The docs state that drift tests do not react to an increase in empty values, and a broken upstream join that nulls half a column is a more common incident than real-world drift.

Step 5: Fail a CI job when drift crosses a threshold

The same report becomes a gate: run it in a pipeline (before promoting a retrained model, or nightly against fresh data) and exit non-zero on any failed test. This script uses Reference(relative=0.2) to also check that the batch is not suspiciously small.

# ci_gate.py: usage  python ci_gate.py reference.parquet current.parquet
import sys
import pandas as pd
from evidently import Dataset, DataDefinition, Report
from evidently.presets import DataDriftPreset
from evidently.metrics import MissingValueCount, RowCount
from evidently.tests import lte, gte, Reference

schema = DataDefinition(numerical_columns=["income", "sessions"], categorical_columns=["plan"])
ref = Dataset.from_pandas(pd.read_parquet(sys.argv[1]), data_definition=schema)
cur = Dataset.from_pandas(pd.read_parquet(sys.argv[2]), data_definition=schema)

report = Report([
    DataDriftPreset(num_method="psi", num_threshold=0.2, drift_share=0.5),
    MissingValueCount(column="income", share_tests=[lte(0.01)]),
    RowCount(tests=[gte(Reference(relative=0.2))]),    # within 20% of reference size
], include_tests=True)
snap = report.run(cur, ref)
snap.save_html("drift_report.html")
snap.save_json("drift_report.json")

failed = [t for t in snap.dict()["tests"] if t["status"] == "FAIL"]
for t in failed:
    print("FAIL:", t["name"], "-", t["description"])
print(f"{len(failed)} failed test(s)")
sys.exit(1 if failed else 0)

Against a 2% income shift and a 40% shift:

$ python ci_gate.py reference.parquet current_ok.parquet; echo "exit=$?"
0 failed test(s)
exit=0
$ python ci_gate.py reference.parquet current_bad.parquet; echo "exit=$?"
FAIL: Value Drift for column income - Drift score is 0.43. The drift detection method is PSI. The drift threshold is 0.20.
1 failed test(s)
exit=1

Note the import: the docs page on tests still shows from evidently.future.tests import Reference in places. In 0.7.23 the current path is evidently.tests; the old future path still imports, but I would not build on it. A GitHub Actions workflow around it, with the report uploaded whether the gate passes or not:

# .github/workflows/drift-gate.yml
name: drift-gate
on:
  pull_request:
    paths: ["models/**", "features/**"]
  schedule:
    - cron: "30 5 * * *"          # daily, 05:30 UTC
permissions:
  contents: read
jobs:
  drift:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v7
      - uses: actions/setup-python@v7
        with:
          python-version: "3.12"
          cache: pip
      - run: pip install -r requirements.txt     # pins evidently==0.7.23, pandas==3.0.6
      - run: python scripts/fetch_batches.py     # your code: writes the two parquet files
      - run: python ci_gate.py data/reference.parquet data/current.parquet
      - uses: actions/upload-artifact@v7
        if: always()
        with:
          name: drift-report
          path: drift_report.*

For gating LLM output quality rather than tabular drift, the same pattern applies with descriptors (Step 8). I cover the promptfoo version of that in how to run LLM evals in CI.

Step 6: Schedule drift checks and store snapshots

A CI gate answers "is this batch OK?". Monitoring answers "how has drift moved over the last month?", which needs results stored over time. Evidently stores each run as a JSON snapshot in a workspace: a local directory, an S3-compatible bucket through fsspec, or a SQL database, per the self-hosting docs. Those docs also note that Evidently Cloud is no longer offered as SaaS, so self-hosting is the path.

# daily_drift.py: run once per batch from cron, Airflow or a Kubernetes CronJob
import sys
from datetime import datetime, timezone
import pandas as pd
from evidently import Dataset, DataDefinition, Report
from evidently.presets import DataDriftPreset
from evidently.metrics import MissingValueCount
from evidently.tests import lte
from evidently.ui.workspace import Workspace

WORKSPACE, PROJECT = "workspace", "churn-model"
schema = DataDefinition(numerical_columns=["income", "sessions"], categorical_columns=["plan"])

ws = Workspace.create(WORKSPACE)                 # opens the workspace if it exists
found = ws.search_project(PROJECT)
project = found[0] if found else ws.create_project(PROJECT)

reference = Dataset.from_pandas(pd.read_parquet("reference.parquet"), data_definition=schema)
current = Dataset.from_pandas(pd.read_parquet(sys.argv[1]), data_definition=schema)

report = Report([
    DataDriftPreset(num_method="wasserstein", cat_method="jensenshannon", drift_share=0.5),
    MissingValueCount(column="income", share_tests=[lte(0.01)]),
], include_tests=True)
snapshot = report.run(current, reference, timestamp=datetime.now(timezone.utc))
ws.add_run(project.id, snapshot, include_data=False)   # metrics only, no raw rows
print("logged", len(ws.list_runs(project.id)), "runs to", project.id)

$ python daily_drift.py current_bad.parquet
logged 8 runs to 01a1906f-d76e-7f26-a73b-19b5683371d8

Pass timestamp= explicitly, set to the time the data describes, not the time the job ran. Backfills and late jobs otherwise pile up on one point of the chart. Keep include_data=False unless you need raw rows in the UI; the snapshots stay small and you avoid copying customer data into a monitoring store. On Kubernetes, a batch/v1 CronJob is enough:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: churn-drift
spec:
  schedule: "0 6 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      backoffLimit: 1
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: drift
              image: registry.example.com/ml/drift-job:0.7.23   # your image with requirements.txt
              args: ["python", "daily_drift.py", "s3://ml-batches/churn/latest.parquet"]
              env:
                - name: FSSPEC_S3_ENDPOINT_URL      # only for MinIO or other S3-compatible stores
                  value: "http://minio.minio.svc:9000"

Reading s3:// paths with pandas needs s3fs in the image. The docs show the UI reading a workspace straight from a bucket with evidently ui --workspace s3://my_bucket/workspace plus the FSSPEC_S3_* variables; for the job itself, the simplest reliable setup is a mounted volume or a sync step.

Step 7: Launch the Evidently UI and add a drift dashboard

Start the UI over the workspace:

evidently ui --workspace ./workspace --port 8000
# INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
curl -s http://127.0.0.1:8000/api/projects
# [{"id":"01a1906f-...","name":"churn-model",...}]

The default host is 127.0.0.1, so in a container you must add --host 0.0.0.0 or nothing outside it can connect. The --secret flag protects write operations only (that is how evidently ui --help describes it), so put the UI behind your SSO proxy or keep it on an internal network.

Dashboards are defined as code. This panel plots the share of drifted columns per snapshot. DriftedColumnsCount returns both a count and a share, so metric_labels picks one, as described in the dashboard API docs:

# dashboard.py: run once; panels persist in the workspace
from evidently.ui.workspace import Workspace
from evidently.sdk.models import PanelMetric
from evidently.sdk.panels import DashboardPanelPlot

ws = Workspace.create("workspace")
project = ws.search_project("churn-model")[0]
project.dashboard.add_panel(
    DashboardPanelPlot(
        title="Share of drifted columns",
        size="full",
        values=[PanelMetric(legend="share", metric="DriftedColumnsCount",
                            metric_labels={"value_type": "share"})],
        plot_params={"plot_type": "line"},
    ),
    tab="Drift",                    # created if it does not exist
)

Built-in alerting is a Pro feature of Evidently Cloud and Enterprise, and with Cloud no longer sold as SaaS that means Enterprise, so with open source I alert from the job: export the drifted share to Prometheus or send it to your incident tool when tests fail. If those alerts get noisy, the routing and grouping advice in reducing alert fatigue with Alertmanager applies unchanged.

Step 8: Monitor LLM outputs with descriptors

Free text cannot go through a KS test, so Evidently turns each response into row-level descriptors (length, word matches, regex hits, sentiment, or LLM-judge labels) and then treats those as ordinary columns. Descriptors can carry their own tests, and TestSummary combines them per row, as shown in the descriptors docs. This example needs no API key:

# llm_drift.py: descriptors on LLM answers, then dataset-level tests
import pandas as pd
from evidently import Dataset, DataDefinition, Report
from evidently.descriptors import TextLength, IncludesWords, TestSummary
from evidently.metrics import ValueDrift, CategoryCount
from evidently.tests import lte, eq

last_week = pd.DataFrame({"answer": [
    "Restart the pod with kubectl rollout restart deployment/api.",
    "Check the readiness probe path; it returns 404.",
    "Increase the memory limit to 512Mi and redeploy.",
] * 200})
this_week = pd.DataFrame({"answer": [
    "I'm sorry, I can't help with that request.",
    "Check the readiness probe path; it returns 404. Also consider reviewing logs, events, and metrics, then escalate.",
    "Increase the memory limit to 512Mi and redeploy.",
] * 200})

def to_dataset(df):
    return Dataset.from_pandas(df, data_definition=DataDefinition(text_columns=["answer"]),
        descriptors=[
            TextLength("answer", alias="length"),
            IncludesWords("answer", words_list=["sorry", "can't"], alias="refusal",
                          tests=[eq(False, alias="not a refusal")]),
            TestSummary(success_all=True, alias="passed"),
        ])

ref, cur = to_dataset(last_week), to_dataset(this_week)
report = Report([
    ValueDrift(column="length"),
    CategoryCount(column="refusal", category=True, share_tests=[lte(0.05)]),
])
snap = report.run(cur, ref)

Output:

ValueDrift(column=length,method=chi-square p_value,threshold=0.05) => 0.0
CategoryCount(column=refusal,categories=True) => {'count': 200.0, 'share': 0.333}
TestStatus.FAIL | Column 'refusal' categories 'True': Less or Equal 0.050

The refusal-rate test catches the regression a model swap or prompt change would cause. Notice also that ValueDrift chose chi-squared for length: the toy data has only three distinct lengths, and numerical columns with five or fewer unique values are treated as categorical. Real traffic will not have that problem, but it is a good reminder to read which method the report picked. For semantic checks, swap in LLM-judge descriptors such as DeclineLLMEval, which need a provider API key and cost tokens per row, so run them on a sample.

Troubleshooting

Symptom	Likely cause	Fix
ImportError: cannot import name 'infer_dtype_from_object'	pandas 3.1.0 with Evidently 0.7.23	Pin pandas==3.0.6 until #1974 is fixed
ModuleNotFoundError: evidently.metric_preset or TestSuite	Tutorial written for the pre-0.7 API	Use Report, Dataset and evidently.presets; see the migration guide
Every batch shows drift on most columns	KS or chi-squared on large samples, or a reference that is not representative	Use num_method="wasserstein" and cat_method="jensenshannon", sample down, rebuild the reference
Dataset drift never fires though key columns drift	Default drift_share=0.5 with many unimportant columns	Restrict columns= to top features or lower drift_share
ValueError: An empty column 'x' was provided for drift calculation	Column entirely null in one dataset	Fix the pipeline; add MissingValueCount tests so nulls fail loudly instead
Half a column went null and drift passed	Drift tests drop nulls before comparing	Add MissingValueCount(column=..., share_tests=[lte(0.01)])
An ID or code column shows huge drift	Inferred column type	Declare types in DataDefinition; leave IDs out
Drift results look inverted	run(reference, current) instead of run(current, reference)	Current first, reference second
UI unreachable from outside a container	Default host is 127.0.0.1	evidently ui --host 0.0.0.0 --workspace /data/workspace
Dashboard panel stays empty	Metric name or label mismatch, or metric not in logged snapshots	Use exact metric names (DriftedColumnsCount) and metric_labels such as value_type

FAQ

What is the default drift test in Evidently?

It depends on column type and reference size. With 1,000 rows or fewer, numerical columns use the KS test and categorical columns use chi-squared (or a Z-test for binary columns), with drift at p below 0.05. Above 1,000 rows, numerical columns use normed Wasserstein distance and categorical columns use Jensen-Shannon distance, with drift at 0.1 or more. Text columns use a domain classifier.

How do I stop Evidently drift false positives?

Switch p-value tests to distance metrics on large batches, limit the preset to the features your model depends on, set per-column thresholds from a few weeks of history, and make sure the reference window is representative. Test nulls separately so data bugs show up as quality failures, not drift.

Is Evidently free for production use?

The library and the self-hosted UI are open source under Apache-2.0. The docs state that the hosted Evidently Cloud is no longer available as SaaS, so production use means running the UI and storage yourself, which is a small Python web service over a directory, bucket or database.

Can Evidently replace SageMaker Model Monitor?

For data and model quality drift, yes, and AWS says as much. SageMaker Model Monitor is closed to new customers, and AWS's replacement samples run Evidently's DataDriftPreset and ClassificationPreset on a schedule with EventBridge, Lambda and MLflow.

When I set this up for a real model, the first thing I check is the reference window, because a bad reference makes every later alert meaningless. Then I check which method the report actually chose for each column, and I add the null and row-count tests before any drift alert is allowed to page anyone. Snapshots go to object storage with include_data=False, and the model ID and version go in the project description, so when a retrain or model swap happens the dashboard line has an explanation next to it. The official references for everything above are the Evidently quickstart, the drift method reference, the Tests docs and the GitHub repository.

Related: How to Use MLflow 3: Tracking, Model Registry and GenAI Tracing, How to Run LLM Evals in CI with GitHub Actions and Model Drift Explained (2026).