How to Set Up SLO Burn Rate Alerts in Prometheus: Sloth, Pyrra (2026)

This guide is for engineers who already run Prometheus and want SLO alerts that page on real budget loss instead of every error spike. By the end you will have SLI recording rules, multiwindow multi-burn-rate alerts that follow the Google SRE workbook, a promtool unit test that proves they fire, the same SLO generated by Sloth or Pyrra so you never hand-maintain the PromQL again, a Grafana view of remaining budget, and Alertmanager routing that separates pages from tickets. It is written against Prometheus v3.15.0, Alertmanager v0.34.1, Sloth v0.16.0, Pyrra v0.10.2 (Helm chart 1.14.0), kube-prometheus-stack 92.10.1 and Grafana 13.2.4, the current releases on 20 October 2026. You need a Prometheus that scrapes a request counter with a status label (or a latency histogram), and for the Kubernetes steps, a cluster with the Prometheus Operator. If you want the reasoning behind SLOs for AI workloads specifically, the companion post on SRE for LLM applications covers TTFT, provider outages and eval-based quality SLIs.

Step 1: Define the SLI, the SLO and the error budget

An SLI is good events divided by total events, as the SRE workbook puts it in Implementing SLOs. Pick one user journey and write it as a ratio. For an HTTP service:

• Availability SLI: requests not answered with 5xx (and, if you like, not 429) divided by all requests.
• Latency SLI: requests faster than a threshold divided by all requests. Use a histogram bucket boundary as the threshold, never a percentile.

The SLO is the target for that ratio over a window. The workbook suggests a four-week rolling window as a general-purpose default; Sloth defaults to 30 days and Pyrra examples use 4w. The error budget is one minus the SLO. Here is what that budget means in time for a 30-day window:

SLO	Error budget	Full-outage minutes per 30 days
99%	1%	432
99.5%	0.5%	216
99.9%	0.1%	43.2
99.95%	0.05%	21.6
99.99%	0.01%	4.32

The rest of this guide uses a service called checkout with a 99.9% availability SLO over 30 days. Before writing any rules, check that the metric exists and has the labels you expect:

# Run in the Prometheus UI. Expect one series per code, for example code="200" and code="500".
sum by (code) (rate(http_requests_total{service="checkout"}[5m]))

Step 2: Write SLI recording rules for every window

Burn-rate alerts compare error ratios over several windows, from 5 minutes to 3 days. Computing a 3-day rate() inside an alert expression on every evaluation is expensive, so precompute each window as a recording rule. This file is plain Prometheus format; the names follow the level:metric:operations convention from the recording rules docs.

# rules/checkout-slo.yaml  (Prometheus v3.15.0)
groups:
  - name: slo-checkout-sli
    interval: 30s
    rules:
      - record: slo:sli_error:ratio_rate5m
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[5m]))
      - record: slo:sli_error:ratio_rate30m
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[30m]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[30m]))
      - record: slo:sli_error:ratio_rate1h
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[1h]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[1h]))
      - record: slo:sli_error:ratio_rate2h
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[2h]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[2h]))
      - record: slo:sli_error:ratio_rate6h
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[6h]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[6h]))
      - record: slo:sli_error:ratio_rate1d
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[1d]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[1d]))
      - record: slo:sli_error:ratio_rate3d
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[3d]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[3d]))
      - record: slo:sli_error:ratio_rate30d
        expr: |
          sum by (service) (rate(http_requests_total{service="checkout",code=~"5.."}[30d]))
          /
          sum by (service) (rate(http_requests_total{service="checkout"}[30d]))

For a latency SLI, swap the numerator for the bucket at your threshold and the denominator for the histogram count, then subtract from one to get an error ratio. Prometheus 3 normalises le labels to float form, so the bucket is le="0.5" or le="1.0", never le="1":

      - record: slo:sli_error:ratio_rate5m
        labels:
          slo: checkout-latency
        expr: |
          1 - (
            sum by (service) (rate(http_request_duration_seconds_bucket{service="checkout",le="0.5"}[5m]))
            /
            sum by (service) (rate(http_request_duration_seconds_count{service="checkout"}[5m]))
          )

The 30-day rule is only for dashboards. If your Prometheus retention is shorter than 30 days, it will silently compute over whatever it has.

Step 3: Do the burn-rate math

Burn rate is how fast you are spending budget relative to the SLO. A burn rate of 1 spends exactly the whole budget by the end of the window. The formula is:

burn_rate        = observed_error_ratio / (1 - SLO)
budget_consumed  = burn_rate * alert_window / slo_period
time_to_exhaust  = slo_period / burn_rate

Example, 99.9% over 30 days (720 h), error ratio 1.44%:
burn_rate        = 0.0144 / 0.001          = 14.4
budget in 1 h    = 14.4 * 1 / 720          = 0.02   (2% of the budget)
time_to_exhaust  = 720 / 14.4              = 50 h

The SRE workbook's Alerting on SLOs chapter walks through six alerting strategies and lands on multiwindow, multi-burn-rate as the recommended one. For a 99.9% SLO it pages when 2% of the budget burns in an hour or 5% in six hours, and opens a ticket when 10% burns in three days. Each long window is paired with a short window one twelfth its length, so the alert stops firing soon after the problem stops (the workbook calls this reset time). The chapter's summary table lists three conditions; its example ticket rule adds a fourth at 3x over 1d/2h, and Sloth generates all four:

Severity	Long window	Short window	Burn rate	Budget consumed	Exhausts budget in
Page	1h	5m	14.4	2%	about 2 days
Page	6h	30m	6	5%	5 days
Ticket	1d	2h	3	10%	10 days
Ticket	3d	6h	1	10%	30 days

Step 4: Write the multiwindow alerting rules

Add an alerting group to the same file. Each alert is two and pairs joined by or, which is the shape of the workbook's own example:

  - name: slo-checkout-alerts
    rules:
      - alert: CheckoutErrorBudgetBurn
        expr: |
          (
            slo:sli_error:ratio_rate1h{service="checkout"} > (14.4 * 0.001)
            and
            slo:sli_error:ratio_rate5m{service="checkout"} > (14.4 * 0.001)
          )
          or
          (
            slo:sli_error:ratio_rate6h{service="checkout"} > (6 * 0.001)
            and
            slo:sli_error:ratio_rate30m{service="checkout"} > (6 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          slo: checkout-availability
        annotations:
          summary: checkout is burning its 30d error budget fast
      - alert: CheckoutErrorBudgetBurn
        expr: |
          (
            slo:sli_error:ratio_rate1d{service="checkout"} > (3 * 0.001)
            and
            slo:sli_error:ratio_rate2h{service="checkout"} > (3 * 0.001)
          )
          or
          (
            slo:sli_error:ratio_rate3d{service="checkout"} > (1 * 0.001)
            and
            slo:sli_error:ratio_rate6h{service="checkout"} > (1 * 0.001)
          )
        for: 15m
        labels:
          severity: ticket
          slo: checkout-availability
        annotations:
          summary: checkout is burning its 30d error budget slowly

Check syntax before loading it:

promtool check rules rules/checkout-slo.yaml
# Expected:
# Checking rules/checkout-slo.yaml
#   SUCCESS: 10 rules found

Then reference the file under rule_files in prometheus.yml and reload Prometheus (send SIGHUP, or POST to /-/reload if you started it with --web.enable-lifecycle). The rules should appear on the Status, Rules page within one evaluation interval.

Step 5: Unit test the alerts with promtool

Burn-rate rules are easy to get subtly wrong (a missing label in by, a threshold off by a factor of 100), and they only fire during incidents, which is the worst time to discover a typo. promtool test rules replays synthetic series through your rules; the format is in the unit testing docs. This test feeds a 2% error ratio (burn rate 20) and expects the page, then feeds 0.05% (burn rate 0.5) and expects nothing:

# tests/checkout-slo-test.yaml
rule_files:
  - ../rules/checkout-slo.yaml
evaluation_interval: 1m
tests:
  - interval: 1m
    input_series:
      - series: 'http_requests_total{service="checkout",code="200"}'
        values: '0+98x120'
      - series: 'http_requests_total{service="checkout",code="500"}'
        values: '0+2x120'
    alert_rule_test:
      - eval_time: 70m
        alertname: CheckoutErrorBudgetBurn
        exp_alerts:
          - exp_labels:
              severity: page
              slo: checkout-availability
              service: checkout
            exp_annotations:
              summary: checkout is burning its 30d error budget fast
          - exp_labels:
              severity: ticket
              slo: checkout-availability
              service: checkout
            exp_annotations:
              summary: checkout is burning its 30d error budget slowly
  - interval: 1m
    input_series:
      - series: 'http_requests_total{service="checkout",code="200"}'
        values: '0+9995x120'
      - series: 'http_requests_total{service="checkout",code="500"}'
        values: '0+5x120'
    alert_rule_test:
      - eval_time: 70m
        alertname: CheckoutErrorBudgetBurn
        exp_alerts: []

promtool test rules tests/checkout-slo-test.yaml
# Expected:
# Unit Testing:  tests/checkout-slo-test.yaml
#   SUCCESS

The ticket alert fires in the first case too, because a 2% error ratio also exceeds the 3x and 1x thresholds on every window that has data. Seventy minutes of synthetic data is enough for rate() to extrapolate over the 1d and 3d windows. Run this in CI on every change to the rules directory.

Step 6: Generate the same SLO with Sloth

Hand-writing ten rules per SLO does not scale past a few services. Sloth takes a short spec and generates the recording rules, metadata rules and both alert severities using the same workbook windows (14.4x and 6x pages, 3x and 1x tickets, as its generated example shows). Download the v0.16.0 binary for your platform from the release page, then write a spec:

# slos/checkout.yaml  (Sloth v0.16.0)
version: "prometheus/v1"
service: "checkout"
labels:
  owner: "payments"
slos:
  - name: "requests-availability"
    objective: 99.9
    description: "Checkout requests that do not return 5xx."
    sli:
      events:
        error_query: sum(rate(http_requests_total{service="checkout",code=~"5.."}[{{.window}}]))
        total_query: sum(rate(http_requests_total{service="checkout"}[{{.window}}]))
    alerting:
      name: CheckoutHighErrorRate
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

sloth validate -i ./slos
sloth generate -i ./slos/checkout.yaml -o ./rules/checkout-sloth.yaml
promtool check rules ./rules/checkout-sloth.yaml

Sloth writes rules named slo:sli_error:ratio_rate5m through slo:sli_error:ratio_rate30d, plus slo:objective:ratio, slo:error_budget:ratio, slo:current_burn_rate:ratio and slo:period_error_budget_remaining:ratio, all labelled with sloth_service, sloth_slo and sloth_id. If you used the hand-written rules from Step 2, delete them before loading Sloth's, or you will have two series with the same name and different labels. The default period is 30 days; pass --default-slo-period=28d to use 28. On Kubernetes, use the sloth.slok.dev/v1 PrometheusServiceLevel CRD (the k8s example uses errorQuery and totalQuery in camelCase) and run Sloth as a controller from its Helm chart, which emits Prometheus Operator PrometheusRule objects.

Step 7: Or use Pyrra on Kubernetes

Pyrra does the same job with a web UI showing budget and burn rate per SLO. It runs as an operator that watches ServiceLevelObjective objects and writes PrometheusRule objects. Install it next to kube-prometheus-stack:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo add pyrra https://pyrra-dev.github.io/helm-charts
helm repo update

helm install kps prometheus-community/kube-prometheus-stack \
  --version 92.10.1 --namespace monitoring --create-namespace

helm install pyrra pyrra/pyrra --version 1.14.0 --namespace monitoring \
  --set prometheusUrl=http://prometheus-operated.monitoring.svc.cluster.local:9090 \
  --set genericRules.enabled=true \
  --set dashboards.enabled=true

Then declare the SLO. The release: kps label matters: kube-prometheus-stack's default rule selector only picks up PrometheusRule objects carrying its release label, and Pyrra copies the SLO's labels onto the rules it generates.

# checkout-slo.yaml  (pyrra.dev/v1alpha1, Pyrra v0.10.2)
apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelObjective
metadata:
  name: checkout-availability
  namespace: monitoring
  labels:
    release: kps
    pyrra.dev/team: payments   # becomes a team="payments" label on the rules
spec:
  target: "99.9"
  window: 4w
  description: Checkout requests that do not return 5xx.
  indicator:
    ratio:
      errors:
        metric: http_requests_total{service="checkout",code=~"5.."}
      total:
        metric: http_requests_total{service="checkout"}

kubectl apply -f checkout-slo.yaml
kubectl -n monitoring get prometheusrules | grep checkout
# Expected: checkout-availability   <age>
kubectl -n monitoring port-forward svc/pyrra 9099:9099

Note that Pyrra's factors differ slightly from the workbook: its source scales windows from a 28-day baseline and uses 14x (1h/5m), 7x (6h/30m), 2x (1d/2h) and 1x (4d/6h), with the alert named ErrorBudgetBurn and long, short, severity and exhaustion labels. For latency, use indicator.latency with a success bucket metric and a total count metric, the same pattern as the Caddy example.

Step 8: Keep SLO definitions portable with OpenSLO

If you want SLO definitions that are not tied to one tool, OpenSLO is a vendor-neutral YAML spec. The site documents v1, v2alpha and the older v1alpha schemas. Sloth reads openslo/v1alpha directly, as its example shows:

apiVersion: openslo/v1alpha
kind: SLO
metadata:
  name: checkout-availability
  displayName: Checkout availability
spec:
  service: checkout
  description: Checkout requests that do not return 5xx.
  budgetingMethod: Occurrences
  objectives:
    - ratioMetrics:
        good:
          source: prometheus
          queryType: promql
          query: sum(rate(http_requests_total{service="checkout",code!~"5.."}[{{.window}}]))
        total:
          source: prometheus
          queryType: promql
          query: sum(rate(http_requests_total{service="checkout"}[{{.window}}]))
      target: 0.999
  timeWindows:
    - count: 30
      unit: Day

sloth generate -i checkout-openslo.yaml produces the same rules as Step 6. The newer v1 SLO schema restructures this (indicator.spec.ratioMetric, timeWindow with duration and isRolling, budgetingMethod of Occurrences, Timeslices or RatioTimeslices), so check which version your tool accepts before standardising.

Step 9: Build the Grafana dashboard

With Sloth rules loaded, import Grafana dashboard 14348, which Sloth's README links as its SLO dashboard. With Pyrra, dashboards.enabled=true ships dashboards as ConfigMaps that the Grafana sidecar in kube-prometheus-stack picks up; they need the generic rules enabled above. To build your own panels from the Sloth series:

# Error budget remaining this period (1 = untouched, 0 = spent, negative = overspent)
slo:period_error_budget_remaining:ratio{sloth_service="checkout"}

# Current burn rate (1 = on pace to spend exactly the budget)
slo:current_burn_rate:ratio{sloth_service="checkout"}

# SLI over the last 30 days as a percentage
100 * (1 - slo:sli_error:ratio_rate30d{sloth_service="checkout"})

Put remaining budget as a stat panel with thresholds at 0.5 and 0.25, and burn rate as a time series with a reference line at 1. That pair answers the two questions people actually ask: are we OK this month, and is it getting worse right now.

Step 10: Route pages and tickets in Alertmanager

Send severity="page" to the on-call pager and severity="ticket" to a queue, and suppress the ticket while the page for the same SLO is firing:

# alertmanager.yml  (Alertmanager v0.34.1)
route:
  receiver: default
  group_by: ['alertname', 'service', 'slo']
  routes:
    - matchers: ['severity="page"']
      receiver: oncall-pager
      group_wait: 30s
      repeat_interval: 1h
    - matchers: ['severity="ticket"']
      receiver: team-queue
      group_wait: 5m
      repeat_interval: 24h
inhibit_rules:
  - source_matchers: ['severity="page"']
    target_matchers: ['severity="ticket"']
    equal: ['alertname', 'service', 'slo']
receivers:
  - name: default
  - name: oncall-pager
  - name: team-queue

Fill in the receiver integrations (PagerDuty, Slack, webhook) for your setup and validate with amtool check-config alertmanager.yml.

Troubleshooting

Symptom	Likely cause	Fix
Recording rule returns no data for a latency SLI	Query matches le="1" but Prometheus 3 stores le="1.0"	Use the float form; see the migration guide
Pyrra or Sloth rules never appear in Prometheus	PrometheusRule lacks the release label the operator selects on	Add release: <helm release> to the SLO metadata, or set ruleSelectorNilUsesHelmValues: false
Page fires on a handful of errors at night	Low traffic: two failures in 50 requests is a 4% ratio	Alert only above a minimum request rate, use longer windows, or add synthetic probes
Alert keeps firing an hour after recovery	Only long windows in the expression	Pair every long window with a short one (long / 12)
Alert never fires during a real outage	Errors return 200 with an error body, or the counter stops being scraped	Count failures at the edge; add an absent-metric alert (Pyrra generates one)
Two series with the same rule name and different labels	Hand-written rules and generated rules loaded together	Keep one source of truth per SLO and delete the other file
30d SLI looks too good on a new Prometheus	Retention or data shorter than the window	Raise retention or use Thanos or Mimir for long windows
promtool test rules fails on annotations	Expected annotations do not match the rendered templates	Copy the exact annotation text, or keep annotations static in tests
Every deploy pages	Brief 5xx bursts during rollout exceed the 5m window	Fix the rollout (readiness probes, connection draining) rather than raising thresholds

FAQ

Why is the fast burn rate 14.4?

Because 14.4 times the allowed error rate for one hour spends 2% of a 30-day budget: 14.4 multiplied by 1 hour divided by 720 hours is 0.02. The workbook chose 2% in an hour as the level worth waking someone for. Pyrra rounds to 14 on a 28-day base.

Should I use Sloth or Pyrra?

Both generate correct multiwindow rules. Sloth is a CLI first, fits GitOps pipelines where generated rules are committed and reviewed, and reads OpenSLO. Pyrra is Kubernetes-first with a built-in UI for budget and burn rate. Pick the one that matches how your team already ships Prometheus rules.

How do I alert on a low-traffic service?

Ratios are noisy when the denominator is small. Combine the burn-rate condition with a minimum request rate, generate synthetic traffic so there is always a baseline, or group several small endpoints into one SLO. The workbook's alerting chapter discusses all three.

Do I need a 30-day recording rule for alerting?

No. The alerts only use windows up to three days. The 30-day rule exists for dashboards and budget reporting, and it is the most expensive rule in the file, so evaluate it less often if it becomes a problem.

What I check first in production is the boring part: does promtool test rules run in CI, does each SLO have exactly one source of truth, and does the page route actually reach a human. The math is the easy bit; the incidents I see come from rules that were never loaded or alerts nobody receives. If your next problem is too many alerts rather than too few, the Alertmanager alert fatigue guide is the natural follow-on, and the observability cost post covers what all these series cost you. Official docs: SRE workbook, Alerting on SLOs, Prometheus recording rules, promtool unit testing, Sloth, Pyrra and OpenSLO.

Related: Observability in 2026: OpenTelemetry Graduated, the Collector Is Still v0.162, eBPF Instrumentation Is v0.14, and Datadog Bills $1.12B a Quarter, AIOps in 2026: 47% on ITBench, 10% on Hard On-Call, $6.50 an Investigation, and Four Outages Where the Automation Was the Incident and Agent Observability in 2026: The Spans Are Standard, the Standard Isn't Stable, and Nobody Reads the Logs.