SRE for LLM Apps in 2026: SLOs for TTFT, Provider Outages and Evals

The first time someone asked me for an SLO on an LLM feature, I wrote "99.9% of requests return 200" and felt pleased with myself for about a week. Then a provider started returning fluent, confident, wrong answers with perfect status codes, and the dashboard stayed green while users filed tickets. That is the core problem with SRE for LLM applications: the old tools still work, but the obvious SLI measures the wrong thing. This post goes through how I would set SLOs for an LLM app in 2026: which latency numbers matter (TTFT and time per output token, not average latency), what availability means when your model sits behind someone else's API, how to turn evals into a quality SLI, and how error budgets, fallbacks and rate limits fit together, using the Google SRE book concepts nearly unchanged.

What changes when the service is nondeterministic?

Less than people expect. The SRE workbook defines an SLI as good events divided by total events, and that definition does not care whether the service is a database or a transformer. What changes is the definition of "good".

For an LLM call, a 200 can carry a useless answer, and the same prompt can produce different answers on consecutive calls, and not only because of sampling temperature. Thinking Machines Lab showed in September 2025 that sending one prompt 1,000 times to Qwen3-235B at temperature 0 produced 80 distinct completions, all identical for 102 tokens and then diverging. The cause was kernels that are not batch-invariant: the batch size depends on server load, so your output depends on other people's traffic. With batch-invariant kernels all 1,000 matched, at a performance cost.

So you cannot write an SLO that says "the answer is correct". You can write one that says "X percent of sampled answers pass a defined check". SRE was always about distributions; LLMs just make it impossible to pretend otherwise.

The other change is dependency structure. Most LLM apps rent the model, so availability is capped by a provider whose rate limits follow your billing tier and whose models retire on its schedule. That pushes reliability work into the layer you own: the gateway, retries, fallbacks and the budget policy.

Which latency SLIs matter: TTFT, TPOT or end-to-end?

Average request latency is close to meaningless for LLM traffic, because a 20-token answer and a 2,000-token answer are different workloads. Split it the way the inference engines do. NVIDIA's benchmarking docs define:

• Time to first token (TTFT): from request submission to the first non-empty token. It includes queueing, prefill and network.
• Inter-token latency (ITL, often called TPOT): the average gap between output tokens after the first, computed as (e2e latency minus TTFT) divided by (output tokens minus 1).
• End-to-end latency: roughly TTFT plus output tokens times ITL.

For a streaming chat UI, TTFT is the pause users feel, and ITL decides whether text streams faster than people read. End-to-end matters when a machine waits for the whole response, such as an agent step, and there you should bucket it by output length or you will alert every time someone asks for an essay.

The metric names are converging. OpenTelemetry's GenAI conventions define gen_ai.server.time_to_first_token and gen_ai.server.time_per_output_token histograms for model servers, with recommended bucket boundaries from 1 ms to 10 s for TTFT, though the whole document is still marked Development status and has moved to its own repository. If you self-host on vLLM (v0.32.0 at the time of writing), the server already exports vllm:time_to_first_token_seconds, vllm:inter_token_latency_seconds and vllm:e2e_request_latency_seconds as Prometheus histograms, with a 1.0 s TTFT bucket in the default boundaries. Prefill and decode now behave like different machines, which is why TTFT and ITL need separate SLOs.

Write latency SLOs as thresholds, not percentiles, ideally two thresholds to catch the tail as the workbook suggests. "95% of first tokens within 1 s, 99% within 2.5 s" counts good and total events, so it can feed an error budget and a burn-rate alert. A p95 cannot.

How do you measure availability across LLM providers?

Start with what providers publish. I pulled the last 50 incidents from Anthropic's status page JSON on 20 October 2026. They span 27 July to 7 October 2026, about 72 days. Thirty-two listed the Claude API as an affected component, and 15 of those were rated major or critical. The figure below plots how long each of those 15 stayed open.

That is not "Anthropic was down 22 hours": most were elevated errors on some models, and incident duration is not error rate. OpenAI's status page tells a similar story at a different granularity: 25 incidents between 5 and 19 October 2026, most of them in ChatGPT and Codex products, a few in the API (PDF processing in the Responses API, platform login). A status page measures the provider's view, per product and per model. Your SLI has to measure your users' view.

Two practical consequences. First, measure availability at your own gateway or client, as the share of user requests that got a usable response. "Usable" has to include the streaming case: Anthropic's error docs note that an error can arrive mid-stream after the API has already returned 200, so a status-code SLI undercounts failures on exactly the long responses users care most about. Second, track availability per model, because incidents are per model. Of the 15 above, seven named specific models in the title rather than the whole API or "multiple models".

Then do the composition arithmetic. Your stack at 99.9% in series with one provider model at 99.9% gives about 99.8%. A second, independent provider improves that a lot on paper, but failures correlate (shared cloud regions, the same traffic spike) and the fallback model behaves differently, so fallbacks belong in the error budget design, not on the free-availability shelf.

How do you turn evals into a quality SLI?

The strongest argument for a quality SLI is Anthropic's own September 2025 postmortem. Three infrastructure bugs degraded Claude's output between August and early September 2025: a routing error that at its peak hit 16% of Sonnet 4 requests on 31 August, a TPU misconfiguration that inserted stray Thai or Chinese characters, and a compiler bug that could drop the highest-probability token. None was an availability failure. Anthropic wrote that its standard benchmarks and safety evaluations did not capture the degradation, and one fix was continuous quality evaluation in production. If the provider's evals missed it, your HTTP dashboard will too.

A workable quality SLI looks like this:

1. Sample a fixed fraction of production requests (respecting your privacy rules) into an async pipeline.
2. Score each with cheap deterministic checks first: valid JSON against the schema, required fields present, no refusal string, citations resolve, output language matches input.
3. Run an LLM judge or a reference comparison only on the subset where deterministic checks cannot decide.
4. Emit two counters, evaluated_total and eval_passed_total, labelled by route and model.

Now "98% of sampled responses pass" is a good-over-total ratio. Be honest about the judge, though. In MLOps in 2026 I went through OpenAI's GDPval numbers: its automated grader agreed with expert graders 66% of the time, against 71% for experts agreeing with each other. So set quality SLOs on the deterministic checks with confidence, and treat judge-based SLIs as ticket-level signals rather than pages until you have measured judge agreement on your own data. The agent observability post covers how to get the traces into the pipeline.

Here is what the three SLI families look like as Prometheus recording rules, assuming a self-hosted vLLM backend plus a gateway and eval pipeline that export their own counters. The gateway metric names are mine; the vLLM ones are real.

# llm-sli-rules.yaml  (Prometheus v3.15.0, vLLM v0.32.0)
# Prometheus 3 normalises le labels to float form, so match le="1.0", not le="1".
groups:
  - name: llm-sli
    interval: 30s
    rules:
      # Latency SLI: share of requests with first token within 1 s (vLLM server-side).
      - record: llm:ttft_good:ratio_rate5m
        expr: |
          sum by (model_name) (rate(vllm:time_to_first_token_seconds_bucket{le="1.0"}[5m]))
          /
          sum by (model_name) (rate(vllm:time_to_first_token_seconds_count[5m]))

      # Streaming SLI: share of decode steps under 100 ms (roughly 10 tokens/s or faster).
      - record: llm:itl_good:ratio_rate5m
        expr: |
          sum by (model_name) (rate(vllm:inter_token_latency_seconds_bucket{le="0.1"}[5m]))
          /
          sum by (model_name) (rate(vllm:inter_token_latency_seconds_count[5m]))

      # Availability SLI at the gateway: usable responses, including mid-stream failures.
      # llm_gateway_requests_total{outcome="ok|fallback_ok|error|stream_error|rate_limited"}
      - record: llm:availability_error:ratio_rate5m
        expr: |
          sum by (route) (rate(llm_gateway_requests_total{outcome=~"error|stream_error|rate_limited"}[5m]))
          /
          sum by (route) (rate(llm_gateway_requests_total[5m]))

      # Quality SLI from the async eval pipeline (deterministic checks only).
      - record: llm:quality_error:ratio_rate1h
        expr: |
          1 - (
            sum by (route) (rate(llm_eval_passed_total{checker="deterministic"}[1h]))
            /
            sum by (route) (rate(llm_evaluated_total{checker="deterministic"}[1h]))
          )

The quality rule uses a one-hour window because sampled eval traffic is thin.

A practical SLI menu for LLM applications

SLI	Good event	Where to measure	Basis
Availability	Usable response, no mid-stream error, not a 429/529	Your gateway or client	Claude API errors
TTFT	First non-empty token under threshold (two thresholds)	Client for UX, server histogram for capacity	NVIDIA metric definitions
ITL / TPOT	Average gap between tokens under threshold	Server histogram	OTel GenAI metrics
End-to-end	Full response under threshold, per output-length class	Gateway, non-streaming routes	SRE workbook latency thresholds
Quality	Sampled response passes deterministic checks	Async eval pipeline	Anthropic postmortem
Determinism (optional)	Replayed golden prompt matches expected structure	Scheduled probe	Thinking Machines

Three or four is plenty for one journey; every SLI is something someone must understand at 3 a.m.

Error budgets with fallbacks and rate limits

The SRE book's central idea holds: 100% is the wrong target, the budget is one minus the SLO, and when it runs out, releases slow or stop. For LLM apps the decisions are about what counts against it.

Fallbacks. If the primary model fails and a fallback answers, did the user get a good event? For availability, yes, and you should count it as good, otherwise you are punishing yourself for the resilience you built. But record the outcome separately (fallback_ok in the rules above) and give the fallback path its own quality SLI, because a smaller or different model may pass schema checks and still answer worse. Put a ticket-level alert on fallback share: quietly serving a large slice of traffic from the fallback is a problem even when availability is green. Gateways like LiteLLM do the routing; my LiteLLM proxy guide covers fallback chains and budgets.

Rate limits. A 429 is a user-visible failure, so it burns availability budget, but it is usually your capacity planning failing rather than the provider. Anthropic's rate limit docs explain that limits are per organisation and per model class, enforced with a token bucket, measured in requests, input tokens and output tokens per minute, and that a sharp increase in usage can trip acceleration limits. Two details matter for SRE work. Cached input tokens do not count toward the input-token limit on most models, so prompt caching is a capacity lever as much as a cost one. And a spend-cap 429 has no retry-after header and keeps failing until the next month, so your retry logic must tell it apart from an ordinary rate limit or it will retry forever. Export the anthropic-ratelimit-*-remaining response headers as gauges and alert on headroom before you hit the wall. A 529 overloaded_error is different again: provider-side capacity, a good trigger for fallback.

Retries. The official SDK retries transient errors twice by default with exponential backoff, per the errors page. Stack that under gateway retries and agent-framework retries and one user request becomes a dozen provider calls during an incident, turning a provider's partial outage into your own rate-limit outage. Pick one layer to own retries.

Burn-rate alerting. Do not page on single errors. Use the workbook's multiwindow, multi-burn-rate scheme: page when 2% of a 30-day budget burns in one hour (burn rate 14.4) or 5% in six hours (burn rate 6), each confirmed by a short window one twelfth as long, and ticket on 10% over three days. The step-by-step Prometheus guide has the rules, the math and the generators. For quality SLIs with thin sample counts, use only the slower windows.

The policy. Write down what happens when the budget is gone, before it is gone. The workbook's examples are a production freeze and reliability work prioritised over features. For an LLM app I would add: no prompt or model changes ship while the quality budget is exhausted, because a prompt change is a deploy, even if it lives in a database row instead of a container image.

FAQ

What is a good TTFT SLO for a chat application?

There is no universal number; it depends on prompt length, model size and hosting. Write two thresholds against a distribution you have measured, for example 95% under 1 s and 99% under 2.5 s, and revisit after a month. Measure at the client where possible, since server-side TTFT misses network and gateway time.

Should provider outages count against my error budget?

Yes. Your users do not care whose fault it was, and the budget exists to tell you whether the product is reliable enough. If provider incidents keep exhausting it, that is the signal to invest in fallbacks, caching or a second provider, which is exactly the decision an error budget is supposed to force.

Can you write an SLO for LLM output quality?

Yes, if you define quality as a countable check on sampled traffic rather than "correctness". Deterministic checks such as schema validity, refusals and citation resolution make solid SLIs. LLM-judge scores are useful but noisier, so start them as ticket-level signals.

What is the difference between TPOT and inter-token latency?

They usually mean the same thing: the average time between output tokens after the first. Some tools include TTFT in the average and some do not, so NVIDIA's docs recommend comparing numbers only when the definitions match.

This quarter I would do three things on any LLM feature I ran: move availability measurement to the gateway with mid-stream errors counted, put TTFT and quality SLOs on the one user journey that matters most, and write the error budget policy, including the rule that prompt changes freeze with everything else. None of that needs a new tool. It needs the same discipline we applied to every other dependency we did not control, applied to the one that now writes the answers.

Related: Agent Observability in 2026: The Spans Are Standard, the Standard Isn't Stable, and Nobody Reads the Logs, MLOps in 2026: Evals Are the New Unit Tests, Judges Agree With Experts 66% of the Time, and OpenAI's Evals API Shuts Down on 30 November and AIOps in 2026: 47% on ITBench, 10% on Hard On-Call, $6.50 an Investigation, and Four Outages Where the Automation Was the Incident.