Golden Signals vs RED vs USE Metrics Explained (2026): PromQL Examples

The first dashboard I open during an incident is never the one with forty panels. It is the one with four rows: how many requests, how many failed, how long they took, and how full the thing underneath is. Those four rows are the golden signals, and two other frameworks, RED and USE, are cousins of the same idea. This post explains what each of the three measures, why RED belongs to services and USE belongs to resources, how to write each signal in PromQL, why histograms beat summaries for latency, what changed now that native histograms are stable in Prometheus 3, and how to turn the signals into SLIs and dashboards that help someone at 3 a.m. rather than impress someone in a review.

What are the four golden signals?

The term comes from the Monitoring Distributed Systems chapter of Google's SRE book, which says that if you can only measure four metrics of a user-facing system, measure these:

• Latency: the time it takes to service a request. The chapter insists you separate the latency of successful requests from failed ones, because a fast 500 drags your average down and makes things look healthier than they are, and a slow error is worse than a fast one.
• Traffic: demand on the system in a system-specific unit. HTTP requests per second for a web service, transactions per second for a database, sessions for a streaming service.
• Errors: the rate of requests that fail, explicitly (a 500), implicitly (a 200 with the wrong content) or by policy (anything slower than your promise).
• Saturation: how full the service is, with emphasis on the most constrained resource. Memory for a memory-bound service, I/O for a disk-bound one.

Saturation is the odd one out. The first three describe requests; saturation describes the machine. That split is exactly why the other two frameworks exist.

RED vs USE: services vs resources

The RED Method was created by Tom Wilkie in 2015 for microservices: for every service, track Rate (requests per second), Errors (how many of those fail) and Duration (how long they take). It is the golden signals minus saturation, and the point of dropping saturation was consistency. If every service exposes the same three things, every service dashboard looks the same, and an on-call engineer who has never seen a service can still read its dashboard.

The USE Method is Brendan Gregg's, first published in ACM Queue in 2012, and it is stated in one line: for every resource, check Utilization, Saturation and Errors. Utilization is the average time the resource was busy, saturation is the work it has queued and cannot yet service, errors is the count of error events. A resource is anything with finite capacity: CPUs, memory, disks, network interfaces, but also connection pools, thread pools and queues.

Wilkie's own framing is that RED and USE are two views of the same system. RED tells you the customer is unhappy. USE tells you which part of the machine is responsible.

Framework	Signals	Applies to	Origin	Source
Four golden signals	Latency, traffic, errors, saturation	User-facing systems	Google SRE book	sre.google
RED	Rate, errors, duration	Every request-driven service	Tom Wilkie, 2015	Grafana Labs
USE	Utilization, saturation, errors	Every resource with finite capacity	Brendan Gregg, 2012	brendangregg.com

Which should you use: golden signals, RED or USE?

All of them, at different layers, and the choice is less philosophical than it looks. My rule of thumb:

• Anything that receives requests (an HTTP API, a gRPC service, a queue consumer, an ingress controller, a database seen from its clients) gets RED. One identical row per service.
• Anything with finite capacity (nodes, pods against their limits, disks, connection pools, worker pools, Kafka partitions) gets USE.
• The golden signals are what you get when you put a service's RED row on top of the USE row for the resource that constrains it. That is the on-call overview.

Batch jobs are the awkward case, since they have no request rate. For a CronJob, rate becomes runs per hour, errors becomes failed runs, and duration becomes run time, with a freshness signal (time since last success) on top. The Prometheus instrumentation guide treats offline and batch jobs as their own category for the same reason.

RED and USE in PromQL

Here is a rule file I would start from. It assumes a service instrumented with the OpenTelemetry HTTP semantic conventions, where http.server.request.duration is a stable histogram in seconds with http.route and http.response.status_code attributes, which typically lands in Prometheus as http_server_request_duration_seconds. The USE half uses node_exporter (v1.12.1 is current), whose pressure collector is enabled by default on Linux and exposes pressure stall information, the most honest saturation signal the kernel offers.

# rules/red-use.yaml, for Prometheus 3.16 and node_exporter 1.12
groups:
  - name: red
    rules:
      # Rate: requests per second per service and route
      - record: service_route:http_requests:rate5m
        expr: sum by (job, http_route) (rate(http_server_request_duration_seconds_count[5m]))
      # Errors: share of requests that returned a 5xx
      - record: service:http_errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_server_request_duration_seconds_count{http_response_status_code=~"5.."}[5m]))
          /
          sum by (job) (rate(http_server_request_duration_seconds_count[5m]))
      # Duration: p99 from a classic histogram (needs the le label)
      - record: service:http_latency_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (job, le) (rate(http_server_request_duration_seconds_bucket[5m])))
  - name: use
    rules:
      # Utilization: share of CPU time not idle, per node
      - record: instance:cpu_utilization:ratio_rate5m
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))
      # Saturation: share of time at least one task waited for CPU (PSI)
      - record: instance:cpu_pressure_waiting:ratio_rate5m
        expr: rate(node_pressure_cpu_waiting_seconds_total[5m])
      # Saturation for containers: share of CFS periods that were throttled
      - record: container:cpu_throttled:ratio_rate5m
        expr: |
          sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total[5m]))
          /
          sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total[5m]))
      # Errors: network receive errors per second
      - record: instance:network_receive_errors:rate5m
        expr: sum by (instance) (rate(node_network_receive_errs_total[5m]))

Two notes. First, CPU utilization at 60 percent tells you less than it seems; a node can be 60 percent busy and still have runnable tasks queuing on one hot core, which is what the pressure metric catches. Second, container throttling is the saturation signal people miss on Kubernetes, because a pod can be throttled while the node looks idle. My requests and limits explainer goes into why. Recording rules like these also keep dashboards fast and protect you from the cardinality trap of putting http_route and pod in the same query; if that has already bitten you, see fixing Prometheus high cardinality.

Histograms vs summaries for latency

Duration is where most instrumentation goes wrong, and the choice is between a histogram and a summary. The Prometheus histograms and summaries page lays out the trade-off:

Property	Summary	Classic histogram	Native histogram	Source
Aggregate across pods	Not aggregatable	Yes, if bucket boundaries match	Yes, bucket layouts reconcile automatically	Prometheus docs
Client cost	Relatively expensive (streaming quantiles)	Cheap, counter increments	Cheap, counter increments	Prometheus docs
Quantiles chosen	At instrumentation time	At query time	At query time	Prometheus docs
Series per metric	One per quantile	One per bucket	One composite series	Prometheus docs

The row that matters is the first. A summary computes p99 inside each process, and you cannot average p99s from ten pods into a fleet p99; the maths does not work. For any service with more than one replica, which is every service, use a histogram.

Classic histograms make you choose bucket boundaries up front, and the quantile you get back is only as precise as the bucket it falls into. If your SLO threshold is 300 ms and you have no bucket at 0.3, you cannot answer "what fraction was under 300 ms" exactly. The OpenTelemetry convention ships a sensible default for HTTP servers, [0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10] seconds, and the SRE book suggests roughly exponential boundaries. Either way, put a bucket exactly on every threshold you promise.

Native histograms in Prometheus 3: what changed

Native histograms remove the bucket-choosing problem. Instead of one series per bucket, a native histogram is a single series of composite samples with exponential buckets created on demand, so resolution is high where your data actually is. Prometheus v3.8.0, released on 2 December 2025, was the first release with native histograms as a stable feature, with a catch: scraping them is opt-in through a new scrape_native_histograms setting, and from v3.9 the old --enable-feature=native-histograms flag does nothing. The current release is v3.16.0, from 5 November 2026.

Per the configuration reference, scrape_native_histograms defaults to false globally and per job. Turning it on also changes the default scrape_protocols to put PrometheusProto first, because the text formats most clients emit cannot carry native histograms. The histograms and summaries page notes that client support is still mainly protobuf-based, for example the Go and Java libraries. In Go's client_golang (v1.25.0), setting NativeHistogramBucketFactor on a histogram, such as 1.1 for buckets about 10 percent wide, makes it emit both forms. OpenTelemetry exponential histograms arrive as native histograms automatically.

During a migration I keep both forms with always_scrape_classic_histograms: true until every dashboard and alert has moved. The queries get simpler, because there is no le label and no _bucket suffix:

# p99 latency from a native histogram: no le, no _bucket
histogram_quantile(0.99, sum by (job) (rate(http_server_request_duration_seconds{job="checkout"}[5m])))

# Latency SLI: fraction of requests that finished within 300 ms
histogram_fraction(0, 0.3, sum(rate(http_server_request_duration_seconds{job="checkout"}[5m])))

histogram_fraction is the quiet win. With classic histograms the 300 ms SLI only works if a bucket sits exactly on 0.3; with native histograms you can ask about any threshold after the fact, within the configured resolution. If you cannot change instrumentation, convert_classic_histograms_to_nhcb stores classic histograms as native histograms with custom buckets, which the same page says are much more efficient to store than classic histograms, though precision stays at your original boundaries.

From golden signals to SLIs

An SLI is a golden signal written as a ratio of good events to all events, which is why RED maps onto SLIs so cleanly. Availability is one minus the error ratio from the rule file. Latency is the histogram_fraction query above, or with classic histograms the le="0.3" bucket rate divided by the count rate. Saturation does not make a good SLI, since users do not experience CPU; it is a cause, and it belongs on the dashboard, not in the SLO.

Once you have the ratio, alert on how fast you burn error budget rather than on raw thresholds. The SRE workbook's alerting on SLOs chapter recommends, for a 99.9 percent SLO, paging when 2 percent of the budget burns in one hour (a burn rate of 14.4, checked over 1 hour and 5 minutes) or 5 percent in six hours (burn rate 6), and opening a ticket when 10 percent burns in three days. I walk through generating those rules with Sloth and Pyrra in the SLO burn rate guide, and the alert hygiene side is in reducing alert fatigue with Alertmanager.

Dashboards that help on-call

A golden-signals dashboard is useful if a stranger can answer "is it us, and where?" in thirty seconds. What works for me:

1. One row per service, same four panels, same order. Rate, error ratio, p50/p99 latency, saturation of the constraining resource. Consistency matters more than cleverness; it is the whole argument for RED.
2. Plot errors as a ratio, latency as percentiles, never averages. An average latency hides the tail that users feel.
3. Put the SLO threshold on the panel as a threshold line, so nobody has to remember that 300 ms is the promise.
4. Template on data source and service, not on pod. Pods change; the panel should not.
5. Link down, not across. From the service row, link to the USE dashboard for its nodes or pods, then to logs and traces with the same time range.
6. Keep it in Git. A dashboard edited by hand during an incident is a dashboard nobody can review. The companion guide, how to manage Grafana dashboards as code, builds exactly this RED row with the Grafana Foundation SDK and ships it with provisioning, Terraform or Git Sync.

If you are starting from nothing, kube-prometheus-stack gives you node_exporter, cAdvisor and kube-state-metrics, which is most of USE on day one. RED depends on your applications exposing request metrics, and no amount of platform tooling can do that part for you.

FAQ

What is the difference between the golden signals and the RED method?

The four golden signals are latency, traffic, errors and saturation. RED keeps the first three under different names (duration, rate, errors) and drops saturation, so that every service can expose identical metrics. Measure saturation separately with USE on the resources each service depends on.

Is the USE method still relevant for Kubernetes?

Yes. Nodes, pods against their CPU and memory limits, persistent volumes and connection pools are all resources. CPU throttling from container_cpu_cfs_throttled_periods_total and pressure stall metrics from node_exporter are good saturation signals that plain utilization misses.

Should I use a histogram or a summary for latency in Prometheus?

Use a histogram. Summaries cannot be aggregated across instances, so you cannot compute a fleet-wide p99 from them. Native histograms, stable since Prometheus 3.8 and enabled with scrape_native_histograms, also remove the need to choose bucket boundaries up front.

Are native histograms production-ready?

They have been stable since Prometheus v3.8.0 in December 2025, but scraping them is opt-in and client support is still mostly protobuf-based libraries such as Go and Java. Keep classic histograms alongside with always_scrape_classic_histograms until your dashboards, alerts and remote-write backends handle the native form.

This quarter I would do three things: add one recording rule group per framework like the one above, enable scrape_native_histograms on a single Go or Java service with classic histograms still scraped, and replace any dashboard panel that shows average latency with p99 plus the SLO threshold line. None of it needs a new vendor, which is a nice change.

Related: Observability vs Monitoring: What's the Difference in 2026?, How to Set Up SLO Burn Rate Alerts in Prometheus: Sloth, Pyrra (2026) and How to Troubleshoot Linux Performance in 60 Seconds (2026 Checklist).