Distributed Tracing Explained: traceparent, Sampling and Jaeger v2 (2026)

Every incident review I have sat through without traces had the same middle section: each team showing its own dashboard to prove its service was fine. Distributed tracing exists to end that argument: a trace follows one request across every service it touches and shows where the time went. This post explains distributed tracing as it stands in October 2026: what spans and traces are, how the W3C traceparent header carries context between services, why head and tail sampling behave so differently, what changed with Jaeger v2 and Grafana Tempo 3, whether Zipkin is still worth deploying, how to join traces to logs, and what all of this costs once the bill arrives.

What is distributed tracing, and what is a span?

A span is one timed unit of work: an HTTP handler, a database query, a call to a queue. It has a name, a start and end timestamp, a status (unset, OK or error), attributes such as http.route or db.system, and a pointer to its parent span. A trace is the tree of spans that share one trace ID. The root span is usually the first thing that saw the request, often a gateway or frontend, and every downstream call becomes a child.

Metrics tell you that p99 latency went up. Logs tell you what one process said. A trace tells you that the p99 request spent most of its life in a retry loop three services away, which is the question you actually have during an incident. I wrote about where tracing sits next to the other signals in observability vs monitoring; the short version is that traces are the only signal that carries causality across process boundaries by design.

In practice almost everyone instruments with OpenTelemetry now. The SDK creates spans, an exporter sends them over OTLP, usually to an OpenTelemetry Collector, and the Collector forwards them to a backend such as Jaeger, Tempo or a vendor. The protocol and the data model are the stable part. Backends are interchangeable, which is the point.

How does context propagation work? The W3C traceparent header

For spans in different processes to land in the same trace, the caller has to tell the callee which trace it belongs to. That is context propagation, and the wire format is W3C Trace Context, a W3C Recommendation since 23 November 2021. OpenTelemetry uses it by default.

The traceparent header has four dash-separated fields: a version (00), a 16-byte trace ID as 32 hex characters, an 8-byte parent ID (the span ID of the caller) as 16 hex characters, and one byte of flags whose lowest bit means "sampled". The spec's own example is 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. A companion tracestate header carries vendor-specific key-value pairs, capped at 32 entries.

Trace Context Level 2, still a Candidate Recommendation Draft dated 28 March 2024, adds a second flag bit: "random". When set, at least the rightmost 7 bytes of the trace ID must be random, which lets samplers make consistent probabilistic decisions from the ID alone without coordinating.

Propagation is also where tracing most often breaks. A trace splits into fragments when some hop drops the header: an old proxy that strips unknown headers, a message queue where nobody injected context into the message attributes, a thread pool that loses the current context. If your traces look short and disconnected, check propagation before you blame the backend.

Head vs tail sampling: which one should you use?

You rarely keep every trace. The question is when you decide.

Head sampling decides at the root span, usually with a ratio based on the trace ID, and the decision travels downstream in the sampled flag. It is cheap, stateless and works anywhere. The OpenTelemetry sampling docs state the weakness plainly: it cannot guarantee that every trace with an error is kept, because at the root nobody knows yet whether the request will fail.

Tail sampling waits until the trace is (mostly) complete, then keeps it based on what happened: errors, latency above a threshold, a particular customer attribute. The same docs warn that it must run in stateful components that buffer a lot of data and need monitoring. In the Collector this is the tail sampling processor, still beta for traces, whose README says all spans of a trace must reach the same Collector instance. Its defaults are a 30 second decision_wait and 50,000 traces held in memory.

That "same instance" rule is why tail sampling at scale is a two-tier deployment. A first tier of Collectors runs the load balancing exporter, which hashes the trace ID to pick a backend Collector, and a second tier runs the tail sampler. Here is the shape of it, pinned to Collector contrib v0.163.0 (released 16 October 2026):

# Tier 1: otel/opentelemetry-collector-contrib:0.163.0 (DaemonSet or gateway)
# Routes every span of a trace to the same tier-2 pod by hashing trace_id.
receivers:
  otlp:
    protocols:
      grpc: { endpoint: 0.0.0.0:4317 }
exporters:
  loadbalancing:            # newer builds also accept the name load_balancing
    routing_key: traceID
    protocol:
      otlp:
        tls: { insecure: true }   # use real TLS outside a lab
    resolver:
      dns:
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: 4317
service:
  pipelines:
    traces: { receivers: [otlp], exporters: [loadbalancing] }
---
# Tier 2: same image, StatefulSet behind a headless Service
# (otlp receiver, backend exporter and service block omitted for brevity)
processors:
  tail_sampling:
    decision_wait: 10s        # default 30s; shorter = less memory, more cut-off traces
    num_traces: 100000        # traces buffered in memory per pod
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: baseline-10pct
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

Most teams I talk to end up with both: a generous head ratio in the SDK so the Collector is not drowning, then tail policies that guarantee errors and slow requests survive. If you run tail sampling, remember that span-derived metrics such as request rates must be computed before sampling or corrected for it, otherwise your dashboards count only the interesting tenth. Tempo 3.0 added a span_multiplier_key override for exactly this head-sampling correction, according to its release notes.

Jaeger v2, Tempo 3 and Zipkin: which tracing backend in 2026?

The backend market consolidated around OTLP. Here is where the three open source options stood when I checked their release pages this week.

Backend	Latest release	Architecture and storage	Status note
Jaeger	v2.22.0, 6 Oct 2026	Single binary built on the OpenTelemetry Collector; Elasticsearch, OpenSearch, Cassandra, ClickHouse per its support policy	v1 line ended with v1.76.0 on 3 Dec 2025
Grafana Tempo	v3.1.0, 29 Sep 2026 (v3.0.4 patch 16 Oct)	Object storage, new ingest architecture in 3.0 replacing legacy ingesters	TraceQL metrics GA in 3.0, 28 May 2026
Zipkin	3.6.1, 8 Apr 2026	Java server; Elasticsearch 8/9, OpenSearch, Cassandra, MySQL	Maintenance pace; OTLP ingest via zipkin-otel
AWS X-Ray	Managed	AWS-managed	$5 per million traces recorded, first 100,000 free monthly

Jaeger v2 is the biggest change. The CNCF's ten-year retrospective describes it as built on the OpenTelemetry Collector, with the old collection of v1 binaries folded into one, and says v1 would be fully deprecated in January 2026 after a last release in December 2025. The final v1 tag on GitHub is v1.76.0, dated 3 December 2025. If you still run jaeger-agent and jaeger-collector as separate deployments, you are on unpatched software and the migration is mostly config: Jaeger v2 speaks OTLP natively and its config file is a Collector config with Jaeger extensions.

Grafana Tempo takes the opposite storage bet. It indexes very little and keeps traces in cheap object storage, then uses TraceQL to search them. Tempo 3.0 removed the legacy ingesters, made TraceQL metrics generally available, and added per-label cardinality limits on its metrics generator. It is the natural choice if you already run Grafana, Loki and Mimir, and an awkward one if you do not.

Zipkin is not dead, but it is quiet. Release 3.6.0 in March 2026 was mostly dependency and JRE 25 image updates, and the OpenTelemetry project deprecated its Zipkin exporter specification in December 2025, with security fixes to existing exporters until at least December 2026. The recommended path is OTLP into Zipkin's own OTLP endpoint or through the Collector. I would not start a new deployment on it in 2026, and I would not rip out a working one in a hurry either.

How do you correlate traces with logs?

Traces tell you where; logs often tell you why. Joining them is cheap if you do it at write time. The OpenTelemetry logs data model, which is marked stable, has optional TraceId, SpanId and TraceFlags fields on every log record. If your logging library is bridged to the OpenTelemetry SDK, records emitted inside an active span get those fields automatically. If you write JSON to stdout instead, add trace_id and span_id to your log formatter from the current span context.

Once the IDs are there, the UI does the rest: Grafana can jump from a Tempo span to the Loki lines with the same trace ID, and Jaeger and most vendors have equivalent links. Exemplars do the same trick from the metrics side, attaching a trace ID to a histogram bucket so a latency spike on a dashboard links to a real request. If you are setting up Loki for this, the Loki on Kubernetes guide covers the label side, and continuous profiling extends the same correlation down to stack traces.

One caution: a log line carrying a trace ID is only useful if that trace was kept. With 1 percent head sampling, 99 percent of your log links point at nothing. Tail sampling that keeps errors fixes the case you care about most.

What does distributed tracing cost?

Tracing is usually the most voluminous signal per request, because one request produces many spans each carrying attributes. Public prices make the arithmetic concrete.

AWS X-Ray charges $5 per million traces recorded and $0.50 per million retrieved or scanned, with the first 100,000 recorded traces free each month. A single service at a steady 100 requests per second produces about 259 million traces in a 30-day month. Recording all of them would cost roughly $1,300; at 10 percent sampling it is about $129. AWS's newer Transaction Search route instead prices span ingestion at $0.35 per GB for the first 10 TB in its published examples.

Grafana Cloud splits traces into processing from $0.05 per GB, writing from $0.40 and retention from $0.10, with 50 GB a month in the free tier and in the $19 Pro platform fee. Self-hosting moves the cost to storage and engineers: Tempo's object storage is cheap per GB, Jaeger on Elasticsearch or OpenSearch is not.

The lever that matters is the same everywhere: sample deliberately, drop attributes nobody queries, and keep high-cardinality values like user IDs in span attributes rather than turning them into metric labels, which is how tracing projects accidentally create a Prometheus cardinality problem. My notes on observability cost go further into vendor bills.

FAQ

What is the difference between a trace and a span?

A span is one operation with a start time, duration, status and attributes. A trace is the full tree of spans that share a trace ID, representing one request end to end. Every trace has one root span and any number of children.

Is Jaeger v1 still supported?

No. The last v1 release was v1.76.0 on 3 December 2025, and the project said v1 would be fully deprecated in January 2026. Jaeger v2, built on the OpenTelemetry Collector and accepting OTLP natively, is the supported line, at v2.22.0 as of October 2026.

Should I use head sampling or tail sampling?

Start with head sampling in the SDK because it is free to run. Add tail sampling in a Collector tier when you need guarantees such as keeping every error or every request slower than a threshold. Tail sampling needs all spans of a trace on one Collector, so plan for a load balancing tier.

Is Zipkin dead?

Zipkin still ships releases, most recently 3.6.1 in April 2026, but development is mostly maintenance. OpenTelemetry deprecated its Zipkin exporter specification in December 2025 and recommends sending OTLP instead. For new systems, Jaeger v2 or Tempo is the safer default.

If I were setting up tracing this quarter, I would instrument with OpenTelemetry, propagate W3C context everywhere including queues, run a small two-tier Collector with tail policies for errors and slow requests, and send to Tempo if Grafana is already in the building or to Jaeger v2 otherwise. Then I would put trace_id in every log line on day one, because retrofitting it later is the job nobody gets round to. The OpenTelemetry Collector on Kubernetes guide is the practical starting point.

Related: Observability in 2026: OpenTelemetry Graduated, the Collector Is Still v0.162, eBPF Instrumentation Is v0.14, and Datadog Bills $1.12B a Quarter, Agent Observability in 2026: The Spans Are Standard, the Standard Isn't Stable, and Nobody Reads the Logs and It's 2 AM. Do You Know What Your AI Agent Is Doing?