How to Reduce Alert Fatigue with Alertmanager: Step-by-Step (2026)

This guide is for anyone running Prometheus and Alertmanager whose on-call rotation has started ignoring the pager. By the end you will have a measured baseline of how noisy your alerts are, alert rules that fire on symptoms rather than causes, a routing tree with severity-based receivers, grouping and inhibition that turn an outage into one notification, silences and time intervals for planned work, and a checklist for when Alertmanager stops sending anything at all. Everything here is checked against Alertmanager v0.34.1 (17 September 2026) and Prometheus v3.15.0 (25 September 2026). You need a working Prometheus that already sends alerts to Alertmanager, and amtool, which ships in every Alertmanager release and in the container image.

You are not alone in this. Grafana Labs' 2026 Observability Survey of 1,363 practitioners found alert fatigue was the most cited barrier to faster incident response, at 30 percent, nearly double the next answer. If you are wondering whether an AIOps product would fix this for you, I wrote a plain-English explainer on what AIOps is; the short answer is that the steps below are what those products do first, and you can do them with a YAML file.

How alerts flow through Alertmanager

Before changing anything, it helps to know where each knob sits. Prometheus evaluates rules and sends firing alerts to every Alertmanager. Alertmanager walks each alert down a routing tree, puts it in an aggregation group, and then the notification pipeline applies inhibition, time intervals and silences before deduplicating and sending.

Step 1: Measure how noisy your alerts are

You cannot reduce what you have not counted. Alertmanager exports counters for every notification it attempts and every one it suppresses, and Prometheus keeps a synthetic ALERTS series for every pending or firing alert. Run these in the Prometheus UI and write the numbers down; they are your baseline.

# Notifications actually sent per integration over the last 7 days
sum by (integration) (increase(alertmanager_notifications_total[7d]))

# Notifications suppressed, by reason (silenced, inhibited, muted by time interval)
sum by (reason) (increase(alertmanager_notifications_suppressed_total[7d]))

# Top 10 alerts by time spent firing over 7 days. With a 1m rule
# evaluation interval each sample is roughly one alert-minute.
topk(10, sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[7d])))

# Alerts currently firing, by severity
count by (severity) (ALERTS{alertstate="firing"})

The metric that matters most cannot be scraped: what fraction of pages led a human to do something. Add a field to your incident or on-call tool ("action taken: yes/no") and fill it in for every page. Google's SRE book states the standard bluntly: "Every page should be actionable" (Monitoring Distributed Systems). Any alert that pages and is marked "no action" twice is a candidate for deletion in Step 9.

Step 2: Alert on symptoms, not causes

The single biggest source of noise is paging on causes: high CPU, a full-ish disk, one pod restarting. Each might matter, and most of the time none of them hurt a user. The Prometheus alerting practices page sums it up as "keep alerting simple, alert on symptoms", and for online services suggests alerting on high latency and error rates "as high up in the stack as possible". Causes still belong on dashboards and in tickets; they just should not wake anyone.

Here is the same service alerted two ways. The first rule is the noisy one; the second pages only when users are failing, waits long enough to ignore blips, and holds the alert briefly after recovery so a flapping condition does not resolve and re-fire every minute.

# rules/checkout.yml, Prometheus v3.15.0
groups:
  - name: checkout-symptoms
    interval: 1m
    rules:
      # NOISY (cause-based): delete or demote to a ticket
      # - alert: CheckoutHighCPU
      #   expr: avg by (pod) (rate(container_cpu_usage_seconds_total{container="checkout"}[5m])) > 0.9
      #   for: 5m

      # Symptom: users are getting errors
      - alert: CheckoutErrorRateHigh
        expr: |
          sum(rate(http_requests_total{service="checkout", code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="checkout"}[5m])) > 0.02
        for: 10m               # must be true for 10 minutes before firing
        keep_firing_for: 5m    # stay firing 5m after recovery to stop flapping
        labels:
          severity: page
          team: payments
          service: checkout
        annotations:
          summary: "Checkout 5xx ratio above 2% for 10 minutes"
          runbook_url: "https://runbooks.example.internal/checkout/error-rate"

      # Symptom: users are waiting
      - alert: CheckoutLatencyHigh
        expr: |
          histogram_quantile(0.99,
            sum by (le) (rate(http_request_duration_seconds_bucket{service="checkout"}[5m]))
          ) > 1.5
        for: 15m
        labels:
          severity: ticket
          team: payments
          service: checkout
        annotations:
          summary: "Checkout p99 latency above 1.5s for 15 minutes"

for and keep_firing_for are documented in the Prometheus alerting rules reference; both default to 0s. Validate the file before loading it:

promtool check rules rules/checkout.yml
# Checking rules/checkout.yml
#   SUCCESS: 2 rules found

If you have SLOs, multi-window burn-rate alerts are the grown-up version of this step, and they are covered in the SLO burn-rate alerts guide.

Step 3: Define severity levels and required labels

Routing, grouping and inhibition all key on labels, so make three of them mandatory on every rule: severity, team and service. Keep severities few and defined by the response, not by how scary the alert sounds:

severity	Meaning	Destination	Expected response
page	Users are affected now	PagerDuty / phone	Acknowledge within minutes, any hour
ticket	Will affect users if ignored for days	Ticket queue or team Slack channel	Next business day
info	Context for humans and for inhibition	No notification	None

If you find yourself inventing critical-p2, the problem is the alerts, not the taxonomy.

Step 4: Build the routing tree

The routing tree sends each alert to exactly one receiver unless you set continue: true. The root route must match everything and acts as the catch-all. Child routes are checked in order and the first match wins. This configuration routes pages to PagerDuty, tickets to Slack, and drops info on the floor. Secrets are read from files so they never land in Git.

# alertmanager.yml, Alertmanager v0.34.1
global:
  resolve_timeout: 5m

route:
  receiver: slack-default          # catch-all
  group_by: [alertname, cluster, service]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity="info"
      receiver: blackhole          # never notifies, still usable for inhibition
    - matchers:
        - severity="page"
      receiver: pagerduty-oncall
      group_wait: 10s              # pages go out faster
      repeat_interval: 1h
      routes:
        - matchers:
            - team="payments"
          receiver: pagerduty-payments
    - matchers:
        - severity="ticket"
      receiver: slack-tickets
      repeat_interval: 24h         # nag once a day, not every 4 hours
      active_time_intervals:
        - business-hours

receivers:
  - name: blackhole
  - name: slack-default
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: "#alerts"
        send_resolved: true
  - name: slack-tickets
    slack_configs:
      - api_url_file: /etc/alertmanager/secrets/slack-webhook
        channel: "#alerts-tickets"
        send_resolved: true
  - name: pagerduty-oncall
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-oncall
  - name: pagerduty-payments
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/secrets/pd-payments

time_intervals:
  - name: business-hours
    time_intervals:
      - weekdays: ['monday:friday']
        times:
          - start_time: '09:00'
            end_time: '18:00'
        location: 'Europe/London'

A receiver with no integrations, like blackhole, is valid and is the cleanest way to discard a class of alerts while keeping them visible in the UI and usable as inhibition sources. Note that Slack's send_resolved defaults to false while PagerDuty's defaults to true, per the configuration reference. The active_time_intervals on the ticket route means tickets only notify during business hours; outside them the route still matches and ends route matching as usual, it just sends nothing.

Step 5: Tune grouping so one outage is one notification

Grouping is the cheapest noise reduction there is. Alerts that share the group_by label values are batched into one notification. The three timers, all inherited by child routes unless overridden, do the rest:

• group_wait (default 30s): how long to wait after the first alert in a new group before sending, so related alerts and inhibiting alerts arrive in time. If an alert resolves inside this window, nothing is sent at all.
• group_interval (default 5m): how long before sending an update when new alerts join or resolve. It also sets the timeout for each send, so very small values with a slow receiver will cancel notifications.
• repeat_interval (default 4h): how long before re-sending an unchanged group. It is rounded up to a multiple of group_interval.

Pick group_by labels that describe one problem. [alertname, cluster, service] is a good default: 50 pods of one service failing the same check become one notification, but two different services stay separate. Avoid grouping by instance or pod on page routes, which recreates the flood. The special value group_by: ['...'] disables grouping entirely and is almost never what you want.

Step 6: Add inhibition rules for cascading failures

Inhibition mutes target alerts while a matching source alert is firing, as long as the labels listed in equal have the same values on both. Two rules cover most of the value: a page mutes lower-severity alerts about the same thing, and an infrastructure-level failure mutes everything downstream of it.

# append to alertmanager.yml
inhibit_rules:
  - name: page-mutes-lower-severity
    source_matchers:
      - severity="page"
    target_matchers:
      - severity=~"ticket|info"
    equal: [alertname, cluster, service]

  - name: cluster-unreachable-mutes-cluster
    source_matchers:
      - alertname="ClusterUnreachable"
    target_matchers:
      - alertname!="ClusterUnreachable"
    equal: [cluster]

  - name: dependency-down-mutes-dependents
    source_matchers:
      - alertname="PostgresDown"
    target_matchers:
      - depends_on="postgres"
    equal: [cluster]

Two details from the inhibit_rule reference trip people up. A label missing on both alerts counts as equal, so if you list equal: [cluster] and neither alert has a cluster label, the rule applies globally. And write target and source matchers so no alert matches both sides; the self-inhibition special case is hard to reason about. v0.34.1 also fixed several bugs where inhibited alerts were improperly un-muted, which is reason enough to upgrade if you rely on inhibition.

Step 7: Use silences and time intervals for planned work

Silences are for one-off, time-boxed suppression: a maintenance window, a known issue being fixed. Create them with a comment and an end time, never open-ended.

export ALERTMANAGER_URL=http://alertmanager.monitoring.svc:9093

# Silence one service in one cluster for 2 hours, with a reason
amtool --alertmanager.url=$ALERTMANAGER_URL silence add \
  service="checkout" cluster="prod-eu-1" \
  --duration=2h \
  --comment="CHG-1234 checkout DB failover test"
# prints the silence ID, e.g. 3f1c0e9a-...

# List active silences
amtool --alertmanager.url=$ALERTMANAGER_URL silence query

# End it early when the work is done
amtool --alertmanager.url=$ALERTMANAGER_URL silence expire <silence-id>

To make comments mandatory for everyone using amtool, put comment_required: true in ~/.config/amtool/config.yml or /etc/amtool/config.yml, as described in the amtool README. For recurring windows (nightly batch, weekend deploy freeze), use mute_time_intervals on the route instead of a silence someone has to remember to create. The root route cannot have mute or active intervals; put them on a child.

Step 8: Run Alertmanager in HA without duplicate pages

Alertmanager deduplicates across replicas by gossiping notification state between peers on port 9094, which needs both TCP and UDP open. The rule people break is in the README in bold: "Do not load balance traffic between Prometheus and its Alertmanagers." Prometheus must send every alert to every replica.

# prometheus.yml, Prometheus v3.15.0
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager-0.alertmanager:9093
            - alertmanager-1.alertmanager:9093
            - alertmanager-2.alertmanager:9093

# each replica, Alertmanager v0.34.1
alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/alertmanager \
  --cluster.listen-address=0.0.0.0:9094 \
  --cluster.peer=alertmanager-0.alertmanager:9094 \
  --cluster.peer=alertmanager-1.alertmanager:9094 \
  --cluster.peer=alertmanager-2.alertmanager:9094

Check alertmanager_cluster_members equals your replica count on every pod.

Step 9: Test routing in CI and review alerts monthly

Every config change should be validated and its routing tested before it ships. amtool does both offline, with no running server:

# Syntax and UTF-8 matcher compatibility
amtool check-config alertmanager.yml
# Checking 'alertmanager.yml'  SUCCESS
# Found:
#  - global config
#  - route
#  - 3 inhibit rules
#  - 5 receivers
#  - 0 templates

# Assert that a payments page reaches the payments PagerDuty receiver.
# Exits 1 on mismatch, so it can gate a pull request.
amtool config routes test --config.file=alertmanager.yml \
  --verify.receivers=pagerduty-payments \
  severity=page team=payments service=checkout

# Print the whole routing tree
amtool config routes show --config.file=alertmanager.yml

If check-config logs a warning about the new matcher parser, fix it now: Alertmanager has been moving to UTF-8 strict matchers since v0.27, and the docs say strict mode will become the default. Double-quote matcher values and escape backslashes.

Then put a recurring 30-minute meeting in the calendar, monthly at minimum, weekly while you are recovering from a noisy period. Pull the Step 1 queries and the "action taken" field, and for each alert that paged, decide one of: keep, retune (threshold, for, group_by), demote to ticket, or delete. An alert nobody acts on is a cost with no benefit. The agenda for incidents themselves fits in a blameless postmortem; this meeting is about the alerts.

Troubleshooting

When the problem is the opposite of fatigue, Alertmanager sending nothing, work left to right through the diagram above.

Symptom	Likely cause	Fix
Alert firing in Prometheus, absent in Alertmanager UI	Prometheus has no Alertmanager targets	Check prometheus_notifications_alertmanagers_discovered is above 0 and the alerting block in prometheus.yml
prometheus_notifications_errors_total or _dropped_total rising	Network, TLS or path_prefix mismatch to Alertmanager	Curl http://<am>:9093/api/v2/status from the Prometheus pod; fix scheme, port or prefix
Alert visible in Alertmanager, no notification	Silenced, inhibited or outside an active time interval	amtool alert query -o extended; check alertmanager_notifications_suppressed_total by reason
Notification goes to the wrong receiver	Route order, or a parent route matched first without continue	amtool config routes test --tree with the alert's labels
Slack or PagerDuty sends fail	Bad webhook URL or routing key, or rate limiting	alertmanager_notifications_failed_total by reason: authError is 401/403, rateLimited is 429 (split out in v0.34.0)
Config changes not taking effect	Reload failed, old config still running	alertmanager_config_last_reload_successful is 0; run amtool check-config and fix, then reload via SIGHUP or POST /-/reload
Every alert notifies two or three times	Load balancer in front of the HA cluster, or gossip port blocked	Point Prometheus at every replica; open 9094 TCP and UDP; check alertmanager_cluster_members
No "resolved" messages in Slack	Slack send_resolved defaults to false	Set send_resolved: true on the slack_configs entry
First notification arrives minutes late	for on the rule plus group_wait, or alert joined an existing group and waits for group_interval	Shorten group_wait on the page route; review for; split group_by if unrelated alerts share a group
Notifications cancelled with context deadline errors	group_interval shorter than the receiver's response time	Raise group_interval; check reason="contextDeadlineExceeded" failures

FAQ

What is the difference between a silence and an inhibition in Alertmanager?

A silence is a manual, time-boxed mute that matches labels, created in the UI or with amtool silence add. An inhibition is a standing rule in alertmanager.yml that mutes target alerts automatically while a matching source alert is firing. Use silences for planned work and inhibitions for known cause-and-effect between alerts.

What should group_wait, group_interval and repeat_interval be set to?

The defaults (30s, 5m, 4h) are reasonable starting points. Shorten group_wait to about 10s on page routes so urgent notifications are not delayed, and lengthen repeat_interval on ticket routes to a day so unchanged problems do not nag. Keep repeat_interval a multiple of group_interval, because Alertmanager rounds it up anyway.

How many alerts should page an on-call engineer?

There is no universal number, but the useful test is the ratio, not the count: every page should lead to a human action. If more than a small share of pages in your monthly review end with "no action taken", delete or demote those rules before adding anything new.

Can AIOps or an AI SRE tool fix alert fatigue instead?

They help with correlation once your alerts are clean, but they inherit whatever noise you feed them. Grouping, inhibition and deleting cause-based alerts are what correlation products do first; doing them yourself is free and lets you judge whether a tool adds anything. The AIOps explainer covers what the commercial layers add.

When I take over an alerting setup, the first thing I check is the ratio of pages to actions over the last month, and the second is whether every rule has severity, team and service labels; without those, nothing else in this guide works. On my own platform I keep pages to symptom alerts with runbook links, send everything else to a queue, and run amtool config routes test in CI on every change. The official references are the Alertmanager configuration docs, the Alertmanager concepts page, the Prometheus alerting rules reference and the Alertmanager releases.

Related: AIOps in 2026: 47% on ITBench, 10% on Hard On-Call, $6.50 an Investigation, and Four Outages Where the Automation Was the Incident, Observability in 2026: OpenTelemetry Graduated, the Collector Is Still v0.162, eBPF Instrumentation Is v0.14, and Datadog Bills $1.12B a Quarter and It's 2 AM. Do You Know What Your AI Agent Is Doing?