How to Build Anomaly Detection Alerts in Prometheus 3 (2026)

This guide is for engineers running Prometheus who want alerts that understand "normal for this time of day" instead of a single fixed number. By the end you will have recording rules that compute a z-score and a week-over-week seasonal prediction, an outlier-resistant median-and-MAD variant, the optional double_exponential_smoothing function enabled correctly, Grafana's open-source anomaly rules wired in as an alternative, and promtool unit tests that prove the alerts fire. Everything here targets Prometheus v3.15.1 (21 October 2026), Alertmanager v0.34.2, the kube-prometheus-stack Helm chart 92.16.1 (which ships Prometheus v3.15.1), and grafana/promql-anomaly-detection v0.3.0. You need a working Prometheus with at least a day of data (a week or three for the seasonal rules) and promtool on your path. If you want the background on when anomaly alerts are worth it at all, read the companion post, Anomaly Detection for DevOps Metrics (2026), first.

Step 1: Check your Prometheus version and enable experimental functions

Most of this guide uses stable PromQL: avg_over_time, stddev_over_time, quantile_over_time and offset. Two functions are different. double_exponential_smoothing (called holt_winters in Prometheus 2) and mad_over_time only exist when you pass --enable-feature=promql-experimental-functions (functions docs, feature flags). The 3.0 migration guide says that if you used holt_winters you must both rename it and pass the flag; a query that still says holt_winters will fail to parse.

# Confirm the server version (expect 3.x; this guide was checked against 3.15.1)
prometheus --version

# Plain binary or container: add the flag to the server arguments
prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --enable-feature=promql-experimental-functions

On Kubernetes with kube-prometheus-stack, the chart exposes the operator's enableFeatures field:

# values-anomaly.yaml for kube-prometheus-stack 92.16.1
prometheus:
  prometheusSpec:
    enableFeatures:
      - promql-experimental-functions
    # Seasonal rules look back up to 3 weeks; the chart default is 10d
    retention: 30d

helm upgrade --install kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack \
  --version 92.16.1 -n monitoring -f values-anomaly.yaml

Check retention while you are here. Prometheus defaults to 15 days (storage docs) and the chart sets 10 days. In Prometheus 3.15 the --storage.tsdb.retention.time flag is marked deprecated in favour of storage.tsdb.retention.time in the config file (configuration docs):

# prometheus.yml (Prometheus 3.15+)
storage:
  tsdb:
    retention:
      time: 30d

Step 2: Record a clean input series

Anomaly rules are only as good as their input. Aggregate to the level you want to alert on and record it once, so every later rule reads a cheap, low-cardinality series. If your raw metric has per-pod or per-path labels you do not need, fix that first (How to Fix Prometheus High Cardinality).

# anomaly-base.rules.yml
groups:
  - name: anomaly-base
    interval: 1m
    rules:
      # Per-job request rate in requests per second
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

promtool check rules anomaly-base.rules.yml

Expected output:

Checking anomaly-base.rules.yml
  SUCCESS: 1 rules found

Step 3: Build z-score recording rules and an alert

A z-score says how many standard deviations the current value sits from its mean. The rules below follow GitLab's published approach: a one-week mean and standard deviation, and a threshold of plus or minus 3. I added two guards GitLab's minimal version leaves out: dividing only where the standard deviation is above zero (a flat series would otherwise produce +Inf or NaN), and a minimum-traffic condition so idle jobs cannot alert.

# anomaly-zscore.rules.yml
groups:
  - name: anomaly-zscore
    interval: 1m
    rules:
      - record: job:http_requests:rate5m:avg_over_time_1w
        expr: avg_over_time(job:http_requests:rate5m[1w])

      - record: job:http_requests:rate5m:stddev_over_time_1w
        expr: stddev_over_time(job:http_requests:rate5m[1w])

      # Divide only where stddev > 0; the filter drops flat series
      - record: job:http_requests:rate5m:zscore
        expr: |
          (job:http_requests:rate5m - job:http_requests:rate5m:avg_over_time_1w)
          / (job:http_requests:rate5m:stddev_over_time_1w > 0)

      - alert: RequestRateZScoreAnomaly
        expr: |
          abs(job:http_requests:rate5m:zscore) > 3
          and on (job) job:http_requests:rate5m > 1
        for: 10m
        keep_firing_for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Request rate for {{ $labels.job }} is more than 3 standard deviations from its 1-week mean"

A one-week range at a one-minute rule interval reads about 10,080 samples per series on every evaluation. For a few dozen series that is nothing. For thousands, move the two _1w rules into a group with interval: 5m and read them with last_over_time(...[10m]), which is the pattern Grafana's framework uses for its long-term bands. keep_firing_for stops an alert resolving and re-firing when a value briefly dips back inside the band (alerting rules docs).

Step 4: Check that your data suits a z-score

Z-scores assume something close to a normal distribution. GitLab's post suggests testing that a week of history produces z-scores of about plus or minus 4, and says a range of plus or minus 20 means the tail is too long. Run these two queries in the Prometheus UI:

# Highest z-score seen in the past week, per job
(
  max_over_time(job:http_requests:rate5m[1w])
  - avg_over_time(job:http_requests:rate5m[1w])
) / stddev_over_time(job:http_requests:rate5m[1w])

# Lowest z-score seen in the past week, per job
(
  min_over_time(job:http_requests:rate5m[1w])
  - avg_over_time(job:http_requests:rate5m[1w])
) / stddev_over_time(job:http_requests:rate5m[1w])

If a job's maximum is 15, a 3-sigma alert on it will fire on ordinary peaks. Use the median-and-MAD variant in Step 6 or the seasonal prediction in Step 5 for that job instead. Latency percentiles and error rates are the usual offenders.

Step 5: Add a week-over-week seasonal prediction

A one-week mean flattens the daily cycle: 3am and 3pm get the same baseline. A seasonal prediction asks what this hour looked like on previous weeks. GitLab's rule takes a four-hour window centred on the same time one, two and three weeks ago ([4h] offset 166h covers 170h to 166h ago), adds the growth in the weekly mean since then, and takes the median of the three:

# anomaly-seasonal.rules.yml (needs retention above 506h, i.e. 3 weeks plus 2h)
groups:
  - name: anomaly-seasonal
    interval: 1m
    rules:
      - record: job:http_requests:rate5m_prediction
        expr: |
          quantile(0.5,
            label_replace(
              avg_over_time(job:http_requests:rate5m[4h] offset 166h)
              + job:http_requests:rate5m:avg_over_time_1w
              - job:http_requests:rate5m:avg_over_time_1w offset 1w,
              "offset", "1w", "", "")
            or
            label_replace(
              avg_over_time(job:http_requests:rate5m[4h] offset 334h)
              + job:http_requests:rate5m:avg_over_time_1w
              - job:http_requests:rate5m:avg_over_time_1w offset 2w,
              "offset", "2w", "", "")
            or
            label_replace(
              avg_over_time(job:http_requests:rate5m[4h] offset 502h)
              + job:http_requests:rate5m:avg_over_time_1w
              - job:http_requests:rate5m:avg_over_time_1w offset 3w,
              "offset", "3w", "", "")
          ) without (offset)

      # Traffic fell below half of what this hour usually sees
      - alert: RequestRateBelowSeasonalPrediction
        expr: |
          job:http_requests:rate5m < 0.5 * job:http_requests:rate5m_prediction
          and on (job) job:http_requests:rate5m_prediction > 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} traffic is under 50% of its seasonal prediction"

The label_replace calls keep the three weeks as distinct series so quantile(0.5, ...) can take their median; the median means one bad week (an outage, a holiday) does not drag the prediction. The prediction depends on job:http_requests:rate5m:avg_over_time_1w from Step 3 having existed three weeks ago, so expect it to fill in gradually after you deploy.

Step 6: Smooth noisy series and use a robust z-score

For spiky metrics, two options help. The first is double_exponential_smoothing(v range-vector, sf scalar, tf scalar), where sf is the smoothing factor and tf the trend factor, both between 0 and 1. The docs say it should only be used with gauges, works on float samples, and is "Holt Linear", not the triple-smoothing seasonal Holt-Winters method its old name suggested (functions docs). A recorded rate is a gauge, so this is valid:

# anomaly-robust.rules.yml (requires --enable-feature=promql-experimental-functions)
groups:
  - name: anomaly-robust
    interval: 1m
    rules:
      # De-noised level: low sf weights older data more, tf follows trend
      - record: job:http_requests:rate5m:smoothed
        expr: double_exponential_smoothing(job:http_requests:rate5m[30m], 0.3, 0.1)

      - record: job:http_requests:rate5m:median_1d
        expr: quantile_over_time(0.5, job:http_requests:rate5m[1d])

      - record: job:http_requests:rate5m:mad_1d
        expr: mad_over_time(job:http_requests:rate5m[1d])

      # 1.4826 scales MAD to be comparable with a standard deviation for normal data
      - record: job:http_requests:rate5m:robust_zscore
        expr: |
          (job:http_requests:rate5m:smoothed - job:http_requests:rate5m:median_1d)
          / (1.4826 * (job:http_requests:rate5m:mad_1d > 0))

      - alert: RequestRateRobustAnomaly
        expr: abs(job:http_requests:rate5m:robust_zscore) > 4
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.job }} request rate is far from its 24h median"

The median and median absolute deviation are not dragged by a single spike the way a mean and standard deviation are. The 1.4826 constant is the standard scale factor that makes MAD estimate the standard deviation of normally distributed data (reference). If you would rather not run an experimental flag in production, note that only mad_over_time and the smoothing function need it. quantile_over_time is stable, and Grafana's robust strategy (Step 7) builds its MAD as quantile_over_time(0.5, ...) over recorded absolute deviations, so it gets the same effect without the flag.

Step 7: Use Grafana's promql-anomaly-detection framework instead

If you would rather adopt tested rules than maintain your own, Grafana's promql-anomaly-detection (Apache-2.0) packages the same ideas with fixes for what broke at scale: a smoothed standard deviation, a low-variability filter, margin bands, and a seasonal band from offset 23h30m (Grafana blog), which the v0.3.0 adaptive rules extend with a weekly offset 167h30m (the robust rules use 23h45m and 167h45m). Its adaptive and robust strategies use only stable functions (the robust one is built on quantile_over_time), so no feature flag is needed.

git clone --branch v0.3.0 --depth 1 https://github.com/grafana/promql-anomaly-detection.git
sudo mkdir -p /etc/prometheus/rules
sudo cp promql-anomaly-detection/rules/adaptive.yml promql-anomaly-detection/rules/robust.yml /etc/prometheus/rules/

# prometheus.yml
rule_files:
  - /etc/prometheus/rules/*.yml

Then tag the series you want analysed. Any series with an anomaly_name label is picked up; anomaly_strategy selects adaptive (default) or robust, and anomaly_type adjusts behaviour (requests excludes sparse series under 5 requests a minute; latency and errors only alert on increases):

# /etc/prometheus/rules/anomaly-inputs.yml
groups:
  - name: anomaly-inputs
    rules:
      - record: anomaly:request:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
        labels:
          anomaly_name: "http_requests"
          anomaly_type: "requests"
          anomaly_strategy: "adaptive"

      # The framework treats zeros as missing, so add its documented zero-traffic alert
      - alert: TrafficStopped
        expr: |-
          anomaly:request:rate5m{anomaly_name="http_requests"} == 0
          and min_over_time(anomaly:request:rate5m{anomaly_name="http_requests"}[1h] offset 23h30m) > 0
        for: 15m
        labels:
          severity: warning

The framework's AnomalyDetected alert fires when anomaly:level sits outside anomaly:upper_band or anomaly:lower_band for 5 minutes, with severity: warning. The README warns that most strategies need 24 to 26 hours of data before the bands settle. On kube-prometheus-stack, wrap each group list in a PrometheusRule (apiVersion: monitoring.coreos.com/v1) carrying your release label, because the chart's default rule selector only picks up rules with it.

Step 8: Unit test the rules with promtool and route the alerts

Write a test before trusting any of this. The test below feeds a flat 100 requests per second for 24 hours, then 300 for 30 minutes, and checks that the z-score alert is still pending after 4 minutes and firing after 14 (unit testing docs):

# anomaly-zscore.test.yml
rule_files:
  - anomaly-zscore.rules.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'job:http_requests:rate5m{job="api"}'
        values: '100x1440 300x30'
    alert_rule_test:
      - eval_time: 24h5m
        alertname: RequestRateZScoreAnomaly
        exp_alerts: []
      - eval_time: 24h15m
        alertname: RequestRateZScoreAnomaly
        exp_alerts:
          - exp_labels:
              severity: warning
              job: api
            exp_annotations:
              summary: "Request rate for api is more than 3 standard deviations from its 1-week mean"

promtool check rules anomaly-zscore.rules.yml anomaly-seasonal.rules.yml
promtool test rules anomaly-zscore.test.yml

# Rules that use mad_over_time or double_exponential_smoothing need the flag here too
promtool --enable-feature=promql-experimental-functions check rules anomaly-robust.rules.yml

Then keep anomaly warnings away from the pager. In Alertmanager, send them to a chat receiver with a long repeat interval; reserve paging for SLO burn rates (How to Set Up SLO Burn Rate Alerts in Prometheus):

# alertmanager.yml (Alertmanager 0.34)
route:
  receiver: default
  group_by: ["alertname", "job"]
  routes:
    - matchers:
        - 'alertname=~"RequestRate.*|AnomalyDetected|TrafficStopped"'
        - 'severity="warning"'
      receiver: chat-anomalies
      repeat_interval: 12h
receivers:
  - name: default
  - name: chat-anomalies
    # slack_configs, webhook_configs, etc.

More grouping and inhibition patterns are in How to Reduce Alert Fatigue with Alertmanager.

Troubleshooting

Symptom	Likely cause	Fix
unknown function with name "holt_winters"	Function renamed in Prometheus 3	Rename to double_exponential_smoothing and enable promql-experimental-functions
double_exponential_smoothing or mad_over_time rejected	Feature flag not set on the server or on promtool	Add --enable-feature=promql-experimental-functions to both
z-score series missing for some jobs	Standard deviation is 0 (flat series) and the > 0 filter dropped it	Expected; add a static threshold or use the margin band approach from Grafana's rules
Alert fires every afternoon peak	One-week mean ignores the daily cycle, or data is not normal	Run the Step 4 checks; switch to the seasonal prediction or robust score
rate5m_prediction is empty	Retention shorter than 502h plus the 4h window, or the _1w rule is newer than 3 weeks	Raise retention to 30d and wait for history to accumulate
Spike widens the band so later anomalies are missed	Standard deviation inflated by the outlier	Use median and MAD, or Grafana's smoothed adaptive strategy
Grafana framework never fires on a drop to zero	Zeros are treated as missing data by design	Add the TrafficStopped alert from Step 7
Framework bands too sensitive in the first day	Strategies need 24 to 26 hours to train	Wait a day before routing, or silence AnomalyDetected during warm-up
rule_group_iterations_missed_total increasing	Long-range rules take longer than the group interval	Move _1w rules to a 5m group and read them with last_over_time
Rules load but no PrometheusRule effect on Kubernetes	Missing release label for the chart's rule selector	Add the label the chart expects, or set ruleSelectorNilUsesHelmValues: false

FAQ

Does Prometheus have built-in anomaly detection?

Not as a feature, but PromQL has everything needed. avg_over_time, stddev_over_time, quantile_over_time and offset build z-scores and seasonal predictions as recording rules. Grafana's open-source promql-anomaly-detection packages a tested set of these rules.

Is holt_winters still available in Prometheus 3?

It was renamed to double_exponential_smoothing in Prometheus 3.0 and moved behind --enable-feature=promql-experimental-functions. It is double (Holt linear) smoothing, so it tracks level and trend but not seasonality.

What z-score threshold should I use?

Start at 3 with for: 10m and a minimum-traffic guard, then check a week of history. If ordinary peaks already reach 4 or more, raise the threshold or switch that series to a seasonal or median-based method.

Should anomaly alerts page on-call?

I would not. They flag unusual behaviour rather than user impact, so route them to chat at severity: warning and page on SLO burn rates and hard limits.

What I check first in production is boring: whether the input series is aggregated sensibly and whether retention covers the largest offset in the rules. Most "the anomaly detection is broken" reports I have seen trace back to one of those two, not to the maths. Then I run the Step 4 z-score range check on every series before it is allowed an alert. The official references are the PromQL functions, recording rules, rule unit testing docs and the promql-anomaly-detection README.

Related: How to Set Up SLO Burn Rate Alerts in Prometheus: Sloth, Pyrra (2026), How to Reduce Alert Fatigue with Alertmanager: Step-by-Step (2026) and How to Fix Prometheus High Cardinality: Find and Drop Series (2026)