How to Fix Prometheus High Cardinality: Find and Drop Series (2026)

This guide is for anyone whose Prometheus is slowly eating its memory limit, getting OOMKilled after a deploy, or timing out on dashboards that used to be instant. Most often the cause is cardinality: too many unique label combinations, so too many time series in the head block. By the end you will have found which metrics and labels are responsible, removed or reshaped them with relabel rules, native histograms and recording rules, and put limits in place so the next bad label fails one scrape instead of the whole server. Everything here was checked against the docs and source for Prometheus 3.15.0 (25 September 2026) and applies to the 3.13 LTS line, supported until 31 July 2027. You need HTTP access to the Prometheus UI or API (a kubectl port-forward is fine), jq, and permission to edit the scrape config or your ServiceMonitors.

A quick definition so the rest makes sense. Every unique combination of metric name and label values is one series. http_requests_total{method="GET",path="/users/1"} and {path="/users/2"} are two series. Put a user ID, request ID or raw URL into a label and you multiply series by the number of users, requests or URLs. Prometheus holds recent series in memory, so series count is roughly memory.

Step 1: Confirm the problem is cardinality

Before changing anything, check that series count is actually what moved. These are Prometheus's own metrics, so query them on the server itself (or on whatever scrapes it):

# Series currently in the head block (in memory)
prometheus_tsdb_head_series

# New series created per second; a spike after a deploy is the classic signature
rate(prometheus_tsdb_head_series_created_total[5m])

# Resident memory of the Prometheus process
process_resident_memory_bytes{job="prometheus"}

If prometheus_tsdb_head_series climbed in step with memory, carry on. If series are flat and memory still grows, look at query load instead: one heavy dashboard can exhaust memory on its own, and the --query.max-samples flag (default 50,000,000) is what stops a single query from loading too much.

Port-forward to the server if you run in Kubernetes. The Service name depends on how you installed it; prometheus-operated is what the Prometheus Operator creates:

kubectl -n monitoring port-forward svc/prometheus-operated 9090:9090

Step 2: Find the worst metrics with the TSDB status page

Prometheus already computes the answer for you. In the UI, open Status > TSDB Status. The same data is on the /api/v1/status/tsdb endpoint, which returns headStats, seriesCountByMetricName, labelValueCountByLabelName, memoryInBytesByLabelName and seriesCountByLabelValuePair. Its limit parameter defaults to 10 and accepts up to 10,000.

# Top 15 metric names by series count in the head block
curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \
  | jq -r '.data.seriesCountByMetricName[] | "\(.value)\t\(.name)"'

# Labels with the most distinct values (user_id, path, pod usually show up here)
curl -s 'http://localhost:9090/api/v1/status/tsdb?limit=15' \
  | jq -r '.data.labelValueCountByLabelName[] | "\(.value)\t\(.name)"'

# Total head series, for a before/after number
curl -s 'http://localhost:9090/api/v1/status/tsdb' | jq '.data.headStats.numSeries'

The first list tells you which metric to look at; the second tells you which label is doing the multiplying. A label such as path with tens of thousands of values is almost always a bug, not a feature.

Step 3: Attribute series to jobs, targets and labels with PromQL

Now find out who is sending them. Prometheus records scrape_series_added and scrape_samples_post_metric_relabeling for every target, so this is cheap:

# Targets that created the most new series in their latest scrape
topk(10, scrape_series_added)

# Targets with the most samples per scrape after relabelling, over the last hour
topk(10, max_over_time(scrape_samples_post_metric_relabeling[1h]))

# Sum by job, to see which team or exporter owns the problem
sum by (job) (scrape_samples_post_metric_relabeling)

Then drill into a specific metric. This query is the one people search for, and it is expensive because it touches every series, so run it with a narrow time range and not on a dashboard:

# Series count per metric name (heavy: run ad hoc only)
topk(10, count by (__name__) ({__name__=~".+"}))

# How many distinct values does one label have on one metric?
count(count by (path) (http_requests_total))

# Which values of that label carry the most series?
topk(20, count by (path) (http_requests_total))

The label values API answers the same question without PromQL:

curl -s -G 'http://localhost:9090/api/v1/label/path/values' \
  --data-urlencode 'match[]=http_requests_total' | jq '.data | length'

Step 4: Analyse churn on disk with promtool tsdb analyze

The status page covers the in-memory head. For churn (series that appear and disappear, typically from pod restarts and deploys) look at persisted blocks with promtool tsdb analyze. It ships in the official image. Point it at the data directory; by default it analyses the most recent block, --limit controls list length (default 20), and --match restricts it to one series selector:

# List blocks first, in human-readable form
kubectl -n monitoring exec prometheus-k8s-0 -c prometheus -- \
  promtool tsdb list -r /prometheus

# Analyse the latest block
kubectl -n monitoring exec prometheus-k8s-0 -c prometheus -- \
  promtool tsdb analyze --limit=15 /prometheus

# Only one metric
kubectl -n monitoring exec prometheus-k8s-0 -c prometheus -- \
  promtool tsdb analyze --limit=15 --match='{__name__="http_requests_total"}' /prometheus

The output has a fixed set of sections, taken from the promtool source: block ID, duration, total series, label names, postings counts, then lists titled "Label pairs most involved in churning", "Label names most involved in churning", "Most common label pairs", "Label names with highest cumulative label value length", "Highest cardinality labels" and "Highest cardinality metric names". If pod or container_id tops the churn lists, your problem is restarts multiplied by labels, not a single bad metric. Pod names and image tags are the usual culprits. The analysis reads a block from disk and allocates memory to do it, so on a server already near its limit, copy the block elsewhere and run promtool there.

Step 5: Drop metrics and labels with metric_relabel_configs

The best fix is at the source: remove the label from the instrumentation, or replace raw URL paths with route templates (/users/:id). When you do not own the code, use metric_relabel_configs, which the docs describe as applied to samples as the last step before ingestion. Do not confuse it with relabel_configs, which acts on targets before the scrape and cannot see metric names.

# prometheus.yml (Prometheus 3.15.0 / 3.13 LTS)
scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api:8080"]
    metric_relabel_configs:
      # 1. Drop whole metrics nobody queries
      - source_labels: [__name__]
        regex: "api_debug_.*|http_request_size_bytes_bucket"
        action: drop
      # 2. Remove high-cardinality labels everywhere in this job
      - regex: "request_id|session_id|trace_id"
        action: labeldrop
      # 3. Collapse raw paths into a template (regex is anchored on both ends)
      - source_labels: [__name__, path]
        regex: "http_requests_total;/users/[0-9]+(/.*)?"
        target_label: path
        replacement: "/users/:id$1"
        action: replace

Validate and reload. The reload endpoint needs --web.enable-lifecycle; otherwise send SIGHUP:

promtool check config prometheus.yml
curl -X POST http://localhost:9090/-/reload

Two warnings. First, the docs say to take care that series are still uniquely labelled after labeldrop: if two series differ only by the label you removed, they collide, and you will see prometheus_target_scrapes_sample_duplicate_timestamp_total increase. Aggregate them with a recording rule instead. Second, metric relabelling saves Prometheus memory and storage, but the target still renders and sends everything on every scrape.

With the Prometheus Operator the same rules go in the ServiceMonitor, under spec.endpoints[].metricRelabelings, with camelCase keys (sourceLabels, targetLabel).

Step 6: Switch big histograms to native histograms

Classic histograms are a cardinality multiplier hiding in plain sight: one series per bucket, plus _sum and _count, for every label combination. Twelve bucket series (including +Inf) on a metric with 200 label combinations is 2,800 series once _sum and _count are added. A native histogram stores all buckets in a single series. According to the native histograms specification, the feature became stable in v3.8.0, but scraping still needs an explicit opt-in, and from v3.9 the old native-histograms feature flag is a no-op.

global:
  scrape_native_histograms: true          # opt in (default false)
scrape_configs:
  - job_name: api
    static_configs:
      - targets: ["api:8080"]
    # Keep classic buckets too during the dashboard migration, then remove this
    always_scrape_classic_histograms: true
    # Optional: store classic-only histograms as native histograms with custom buckets
    convert_classic_histograms_to_nhcb: true
    native_histogram_bucket_limit: 160    # cap buckets; resolution is reduced to fit

With scrape_native_histograms: true, Prometheus puts the protobuf format first in its default scrape_protocols, because the text formats cannot carry native histograms. Your client library also has to expose them. In Go's client_golang, that means setting NativeHistogramBucketFactor (for example 1.1) in HistogramOpts. If you remote-write to another system, check that it accepts native histograms and set send_native_histograms: true on the remote write config. Queries change slightly: histogram_quantile(0.99, sum by (le) (rate(x_bucket[5m]))) becomes histogram_quantile(0.99, sum(rate(x[5m]))).

Step 7: Pre-aggregate with recording rules

Recording rules do not reduce what is ingested, but they let dashboards and alerts query a few hundred series instead of a few hundred thousand, and they are what you federate or remote-write upstream instead of the raw data:

# rules/http.yml
groups:
  - name: http_aggregations
    interval: 1m
    rules:
      - record: job_route:http_requests:rate5m
        expr: sum by (job, route) (rate(http_requests_total[5m]))
      - record: job:http_request_duration_seconds:p99_5m
        expr: histogram_quantile(0.99, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))

promtool check rules rules/http.yml

Pair this with a drop rule on the raw metric on any Prometheus that only needs the aggregate, and you have actually removed the series. If you are building SLO alerts on top, the burn-rate alerts guide uses the same pattern.

Step 8: Add limits so the next explosion fails one scrape

Cardinality bugs come back with every new service. Per-scrape limits turn a server-wide outage into one failed target. Each is evaluated after metric relabelling, and exceeding sample_limit, label_limit, label_name_length_limit or label_value_length_limit fails the whole scrape:

scrape_configs:
  - job_name: api
    sample_limit: 50000             # max samples per scrape (0 = no limit)
    label_limit: 30                 # max labels per series
    label_value_length_limit: 200   # bytes; catches URLs and stack traces in labels
    target_limit: 500               # experimental: max targets for this job

Then alert on both the limit and the trend, so a failed scrape gets a human:

groups:
  - name: cardinality-guards
    rules:
      - alert: ScrapeSampleLimitHit
        expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
        labels: { severity: warning }
        annotations:
          summary: "A target exceeded sample_limit; its scrape is failing"
      - alert: HeadSeriesGrowingFast
        # Tune the multiplier to your normal daily variation
        expr: prometheus_tsdb_head_series > 1.3 * (prometheus_tsdb_head_series offset 1d)
        for: 30m
        labels: { severity: warning }

Finally, check the memory behaviour. Prometheus 3 sets GOMEMLIMIT from the container memory limit by default, per the migration guide, with a ratio of 0.9. That makes the garbage collector work harder before the kernel kills the process, but it cannot save a server whose head simply does not fit. Fix the series first, then size memory. If the pod is already crash-looping, the OOMKilled exit code 137 guide covers the Kubernetes side.

Troubleshooting

Symptom	Likely cause	Fix
Prometheus OOMKilled after a deploy	New label with unbounded values (user ID, path, pod hash)	Step 2 to find it, Step 5 labeldrop or drop, then fix the instrumentation
Target shows sample limit exceeded	Scrape exceeded sample_limit after relabelling	Drop metrics on that job or raise the limit deliberately; the whole scrape fails until then
Drop rule has no effect	Rule placed under relabel_configs instead of metric_relabel_configs	Move it; target relabelling cannot see __name__
prometheus_target_scrapes_sample_duplicate_timestamp_total rising	labeldrop made two series identical	Remove the rule, aggregate with a recording rule instead
Memory still high hours after dropping	Old series stay in the head until compaction	Wait for the next head compaction (blocks cut about every two hours); do not restart in a panic
Native histograms never appear	scrape_native_histograms not set, or client does not expose them	Set it globally or per job, enable native buckets in the client library, confirm protobuf is negotiated
Query fails with query processing would load too many samples into memory	Query touches more than --query.max-samples	Narrow the selector or time range, use a recording rule
Series count saw-tooths with every rollout	Churn from pod, instance or version labels	Use promtool tsdb analyze churn lists; aggregate away pod-level labels
Remote write backlog after enabling native histograms	Receiver lacks native histogram support or send_native_histograms not set	Check the receiver's docs, keep classic histograms until it supports them

FAQ

What counts as high cardinality in Prometheus?

There is no fixed number; it is relative to memory. Practically, any single metric with tens of thousands of series, or any label with thousands of distinct values, deserves a look. The TSDB status page ranks both, so start there rather than guessing.

Does metric_relabel_configs reduce memory?

Yes, for Prometheus: dropped samples are never ingested, so they never create head series. It does not reduce the work on the target or the network, because the full payload is still scraped. Already-ingested series leave memory after the next head compaction.

Are native histograms production-ready in 2026?

The Prometheus specification marks them stable since v3.8.0, and both current releases (3.15 and the 3.13 LTS) support them. You still need to opt in with scrape_native_histograms, use a client library that exposes them, and check any remote-write receiver before switching off classic buckets.

Should I move high-cardinality data to traces instead?

Often, yes. Per-request identifiers like user or request IDs belong in span attributes or logs, where the storage model expects them. I cover that split in distributed tracing explained.

When a Prometheus starts falling over, the first thing I check is prometheus_tsdb_head_series against the deploy timeline, then the TSDB status page, because between them they name the metric and the label in under a minute. In production I would put sample_limit and label_value_length_limit on every job from day one and alert on them, since an engineer finding out from a failed scrape is far cheaper than finding out from an OOMKill. The reference pages are the Prometheus configuration docs, the HTTP API docs and the native histograms spec. For what all this costs at vendor prices, see my notes on observability cost.

Related: Observability in 2026: OpenTelemetry Graduated, the Collector Is Still v0.162, eBPF Instrumentation Is v0.14, and Datadog Bills $1.12B a Quarter, AIOps in 2026: 47% on ITBench, 10% on Hard On-Call, $6.50 an Investigation, and Four Outages Where the Automation Was the Incident and It's 2 AM. Do You Know What Your AI Agent Is Doing?.