Anomaly Detection for DevOps Metrics (2026): Thresholds vs Baselines vs ML

Every few months someone asks me to replace our static alert thresholds with "AI anomaly detection" so the pages stop being dumb. A CPU alert at 80% that fires every Monday morning is dumb. But I have also watched anomaly detection for DevOps metrics turn one dumb page into forty clever ones, each with a confidence band and none with a reason to wake up. This post walks through the three families of approach (static thresholds, seasonal baselines, and ML detectors such as Datadog Watchdog, Grafana Machine Learning and Prophet), what each needs in history and maintenance, the false-positive arithmetic vendors leave off the product page, and why I still let SLO burn rates do the paging.

What is anomaly detection for DevOps metrics?

Strip away the marketing and every system answers one question: given what this time series did before, is the current value surprising? The differences are in how "before" is modelled.

• Static thresholds assume nothing about the past. disk_free < 10% is a judgement you wrote down.
• Statistical baselines compute a mean and spread over a window (a z-score), optionally from the same hour last week, and flag values outside a band. You can do all of this in PromQL.
• ML detectors fit a model per series (seasonal decomposition, SARIMA variants, Prophet-style additive models) or compare series against their peers, and then draw a band.

All three output the same thing: a boolean that something is unusual. None of them can tell you whether unusual is bad. That gap is where most of the pain lives.

Static thresholds: why they still catch most real incidents

The Google SRE book is blunt about this. Its chapter on monitoring says the rules that catch real incidents "should be as simple, predictable, and reliable as possible", and that Google avoids "magic" systems that try to learn thresholds (Monitoring Distributed Systems).

Thresholds work on metrics with a physical limit: disk, memory against a cgroup limit, certificate expiry, queue depth against a consumer's capacity, replication lag against an RPO. These do not need a baseline because the failure point does not move with traffic. A queue at 90% of its bound is bad on a Tuesday and bad on Christmas.

Where thresholds fail is on demand-driven metrics. Request rate, CPU on a stateless web tier, and outbound bandwidth all follow the business day. A threshold tight enough to catch a 3am drop will fire every afternoon, so people loosen it until it catches nothing. Prometheus gives you two cheap knobs before you reach for statistics: for, which requires the condition to hold for a duration, and keep_firing_for, which stops flapping and false resolutions (alerting rules docs). Plenty of "we need ML" conversations end at for: 10m.

Seasonal baselines: z-scores and week-over-week bands in PromQL

The best-known public write-up is still GitLab's 2019 post by Andrew Newdigate (Anomaly detection using Prometheus). It records a one-week avg_over_time and stddev_over_time for request rate, computes a z-score, and treats anything beyond plus or minus 3 as anomalous. It also warns that z-scores assume roughly normal data and suggests checking that your historical z-scores sit around plus or minus 4. If they range to 20, the tail is too long and the method is lying to you. Error rates, latencies and queue lengths are often not normal, and the post says those may be better served by fixed thresholds.

For seasonality, GitLab builds a prediction from the same four-hour window one, two and three weeks ago (offsets of 166h, 334h and 502h), adjusts each for growth, and takes the median. That is a decent model of "what Tuesday at 14:00 usually looks like", and it runs inside Prometheus with no extra service. The z-score half fits in a few lines of rules:

# Prometheus 3.15 rule file: one-week z-score on a recorded request rate
groups:
  - name: anomaly-zscore
    rules:
      - record: job:http_requests:rate5m:zscore
        expr: |
          (job:http_requests:rate5m - avg_over_time(job:http_requests:rate5m[1w]))
          / (stddev_over_time(job:http_requests:rate5m[1w]) > 0)  # skip flat series
      - alert: RequestRateZScoreAnomaly
        expr: abs(job:http_requests:rate5m:zscore) > 3
        for: 10m            # must stay anomalous for 10 minutes
        labels:
          severity: warning # never a page on its own

Grafana Labs published a more production-hardened version in 2024 and open-sourced it as promql-anomaly-detection, now at v0.3.0 (15 October 2026, Apache-2.0). The accompanying blog post is the most honest document in this space. It lists what broke in the naive z-score: extreme spikes inflate the standard deviation so later anomalies hide inside a fat band; very stable periods shrink the band to nothing and produce false positives; and recurring spikes such as cron jobs look anomalous every time they run. The fixes are a smoothed one-hour standard deviation averaged over 26 hours, a filter on low coefficient of variation, a minimum-width margin band, and a long-term band from offset 23h30m (the repo's adaptive rules add a weekly offset 167h30m). The repo now ships two strategies, adaptive (mean and standard deviation) and robust (median and MAD), says most strategies need 24 to 26 hours of data, and documents a limitation worth knowing: zero values are treated as missing, so a metric dropping to zero does not trigger AnomalyDetected. You need a separate "traffic stopped" alert for the one anomaly everybody cares about.

I wrote the full rule set up step by step in the companion guide, How to Build Anomaly Detection Alerts in Prometheus 3, including the feature flag that double_exponential_smoothing (the function called holt_winters in Prometheus 2) now needs.

ML anomaly detection: Datadog Watchdog, Grafana ML and Prophet

The commercial tools are better at the boring parts: training, retraining, holidays and per-series parameter choices. They are not magic, and their own documentation says so.

Tool	Method	History needed	What the docs warn about	Source
Prometheus alert rule	Static threshold plus for and keep_firing_for	None	Seasonal metrics need loose thresholds	Prometheus docs
GitLab PromQL rules	1-week z-score; median of 3 prior weeks	3 weeks plus	Assumes normal data; check z range	GitLab blog
Grafana promql-anomaly-detection v0.3.0	adaptive (mean, stddev) or robust (median, MAD) bands	24 to 26 hours	Zeros treated as missing	GitHub
Datadog anomaly monitor	basic, agile (a SARIMA variant), robust (seasonal-trend decomposition)	3 seasons; uses up to 6 weeks	Not for metrics without predictable patterns	Datadog docs
Datadog Watchdog	Built-in baselines across the platform	Not stated	"do not require setup"	Watchdog docs
Grafana ML forecasting	Forecast band, retrained every 24 hours	90 days of training by default	Best for stable trends or seasonality	Grafana docs
Grafana ML outlier detection	DBSCAN or MAD across a group	At least 3 series	Compares peers, not history	Grafana docs
Prophet 1.5.0	Additive model: trend, yearly, weekly, daily seasonality, holidays	Several seasons	Library, not a monitor; you run it	PyPI

A few details matter more than the headline features. Datadog's three algorithms encode a real trade-off: agile adapts quickly to level shifts "at the expense of being less robust to recent, long-lasting anomalies", while robust keeps predictions stable through long anomalies but responds slowly to an intended shift such as a deploy. Weekly seasonality needs at least three weeks of data. Its anomaly monitor FAQ answers "should I use anomaly detection for everything?" with a plain "No" (Datadog guide).

Grafana Cloud's ML features (forecasting, outlier detection and Sift) are free on all Grafana Cloud accounts, and the series they generate do not count as billable active series (Grafana ML docs). Outlier detection answers a different question from forecasting: not "is this pod unusual compared with last week" but "is this pod unusual compared with its siblings right now". For a fleet of identical replicas, that peer comparison is often the most useful anomaly signal you can get, because it ignores the time of day entirely.

Prophet, which shipped 1.5.0 on PyPI on 5 October 2026, is a forecasting library rather than a monitoring product. Its own description says it works best with strong seasonal effects and several seasons of history. Teams use it for capacity forecasts and offline baselines, then write the predicted bands back as metrics. That works, but now you own a batch job, a model store, and the question of what happens when the job is late.

Why anomaly detection produces so many false positives

Here is the arithmetic I wish every vendor demo started with. If a metric's noise really were Gaussian, a two-sided 3-sigma band would be breached by chance about 0.27% of the time. Evaluate one series every minute and that is roughly 3.9 chance breaches a day. Evaluate 1,000 series (a modest Kubernetes cluster's worth of per-service request, error and latency series) and it is about 3,900 a day. Move to 4 sigma and you still get around 90 a day across those 1,000 series. These are textbook normal-distribution numbers I calculated, not measurements, and real metrics are worse because their tails are heavier than Gaussian, which is exactly GitLab's warning.

for: 10m helps less than the maths suggests, because metric noise is autocorrelated: a series that is high this minute is likely high the next. The other lever is base rates. Real incidents on a given series are rare, so even a detector with good recall and a low per-check false-positive rate produces alerts that are mostly wrong. The Google SRE Workbook frames alert design in terms of precision, recall, detection time and reset time (Alerting on SLOs), and anomaly detection on raw metrics tends to buy recall by spending precision. Every false page costs a human context switch at minimum. Spend enough of them and people stop reading the channel, which is how the true positive gets missed. The mechanics of fixing that on the routing side are in How to Reduce Alert Fatigue with Alertmanager.

Should anomaly detection page on-call? SLO burn rates instead

Grafana's own engineers, describing their own framework, wrote that "anomalies aren't strong enough signals on their own to indicate something is wrong" and recommended tying them to SLO-based alerts as part of root-cause analysis. I agree, and it is a nice change to read a vendor arguing against overselling its product.

The paging alternative is the multiwindow, multi-burn-rate alert from the SRE Workbook. For a 99.9% SLO it pages when the error budget burns at 14.4 times the sustainable rate over one hour (confirmed over five minutes), or 6 times over six hours (confirmed over 30 minutes), and opens a ticket at 1 times over three days. Those three rules correspond to spending 2%, 5% and 10% of the monthly budget. A burn-rate alert is anomaly detection with the right definition of anomaly: the deviation that matters is from the promise you made to users, not from last Tuesday. The setup is in How to Set Up SLO Burn Rate Alerts in Prometheus.

My split is simple. Symptoms that users feel page, via burn rates. Physical limits page, via thresholds. Anomaly bands go to a dashboard and a low-priority channel, and get attached to the page as context ("request rate on checkout is 4 sigma above its weekly band") so the human starts the investigation with a hypothesis.

AIOps reality check: does ML anomaly detection beat simple methods?

The research record is humbling. Wu and Keogh's paper (first posted in 2020) argued that most popular time series anomaly benchmarks are flawed and that "much of the apparent progress in recent years may be illusionary" (arXiv:2009.13807). The NeurIPS 2024 TSB-AD benchmark from Liu and Paparrizos evaluated 40 detection algorithms on 1,070 curated series from 40 datasets and found that "simpler architectures and statistical methods often yield better performance", with neural networks showing promise mainly on multivariate data (NeurIPS 2024, TSB-AD repo).

That does not make the ML products useless. Their value is operational: they train and retrain thousands of per-series models, handle daylight saving and holidays, and give you outlier comparison across a fleet without writing rules. It does mean the algorithm is rarely your bottleneck. Label hygiene, sane aggregation, and knowing which metrics are symptoms matter more. That is the same conclusion I reached on AI SRE agents in AIOps in 2026: the model is only as good as the telemetry and the gate in front of the pager. Platform discipline did not become optional because the detector got a neural network.

FAQ

What is the difference between anomaly detection and threshold alerting?

A threshold compares a metric with a fixed value you chose, such as disk free below 10%. Anomaly detection compares it with a band derived from the metric's own history or its peers. Thresholds suit metrics with physical limits; anomaly bands suit demand-driven metrics that follow daily and weekly cycles.

Can Prometheus do anomaly detection without an external tool?

Yes. You can build z-scores with avg_over_time and stddev_over_time, seasonal predictions with offset, and median-based bands with quantile_over_time, all as recording rules. Grafana's open-source promql-anomaly-detection framework packages this into ready-made rules that need 24 to 26 hours of data to settle.

How much history does anomaly detection need?

It depends on the seasonality you model. Datadog asks for three seasons (three weeks for weekly seasonality), GitLab's PromQL prediction uses three prior weeks, and Grafana Cloud forecasting trains on 90 days by default. Brand-new services have no baseline, so give them thresholds and SLOs first.

Should anomaly alerts page the on-call engineer?

Usually not. Anomaly bands flag unusual behaviour, not user harm, and at fleet scale chance breaches add up quickly. Page on SLO burn rates and hard limits, and send anomaly alerts to a triage channel or attach them to the page as context.

This quarter, if I were starting from a noisy alert set, I would do four things in order: put every user-facing service on burn-rate paging, keep thresholds only for real limits, deploy Grafana's PromQL anomaly rules for request rate and latency at severity: warning with a separate zero-traffic alert, and review after a month which anomaly alerts were followed by an incident. If the answer is almost none, the detector is a dashboard, and that is fine.

Related: How to Set Up SLO Burn Rate Alerts in Prometheus: Sloth, Pyrra (2026), What Is AIOps? A Plain-English Explainer for 2026 and Observability vs Monitoring: What's the Difference in 2026?