Structured Logging Best Practices in 2026: JSON, Trace IDs, PII, Cost

Most incidents I have worked in fourteen years reach the same moment: someone greps for a request ID and finds it spelled three ways across three services, one of which logs it inside a sentence. Structured logging is the boring fix, and in 2026 it has converged on a standard shape. This post covers the structured logging best practices I apply: the fields a JSON log line needs, how log levels map to OpenTelemetry severity numbers, trace IDs in every line, what never to log, sampling without losing errors, what logs cost per gigabyte this month, and what changes for LLM apps.

What is structured logging, and why JSON?

A structured log is one event per line, written as key-value data with fixed field names instead of a sentence a human has to parse. JSON is the default encoding because every pipeline parses it without a custom regex. The Go team put the argument well when it shipped log/slog in Go 1.21 in August 2023: structured logs "can be parsed, filtered, searched, and analyzed quickly and reliably."

There is a security argument too. The OWASP Logging Cheat Sheet lists log injection (CWE-117): put a carriage return in a username and you can forge a second log entry. It warns that parameterised logging alone does not stop CR/LF injection; a structured encoder that escapes values does.

Go has slog.NewJSONHandler. Spring Boot 3.4 added structured logging with ECS, GELF and Logstash formats behind logging.structured.format.console. Python has structlog (26.1.0) and Node has pino (v10.4.0).

The rule that matters more than the library: log to stdout, one JSON object per line, and let the platform ship it. On Kubernetes that means a node agent; my Fluent Bit Kubernetes setup guide covers that side.

What fields should every log line have?

Use the OpenTelemetry Logs Data Model as your schema even if you never run an OpenTelemetry SDK. It is Stable, and it settles naming arguments before they start. OWASP's version of the same idea is that every entry records when, where, who and what.

JSON field you emit	OTel LogRecord field	Example	Notes
timestamp	Timestamp	2026-10-24T10:28:35.574871Z	UTC, ISO 8601, sub-second precision
level	SeverityText and SeverityNumber	info / 9	Map to the numeric ranges below
event or message	EventName or Body	payment.authorised	A stable event name beats a sentence
trace_id, span_id, trace_flags	TraceId, SpanId, TraceFlags	32 and 16 lowercase hex chars	Top-level keys, per OTel's JSON convention
service, version, env	Resource (service.name etc.)	checkout	Same for every line from a process
everything else	Attributes	order_id, tenant, amount_minor	Per-event detail, consistent names

Sources: OpenTelemetry Logs Data Model (Stable), OpenTelemetry trace context in non-OTLP logs (Stable), OWASP Logging Cheat Sheet.

Two habits pay off. Give events a stable name (payment.authorised) and keep the sentence optional: you can count and alert on a name, while a sentence drifts with every edit. And put units in field names (duration_ms, size_bytes), so nobody has to read the source to know whether duration is seconds.

Log levels: which severity to use, and the OpenTelemetry numbers

OpenTelemetry defines SeverityNumber from 1 to 24 in six bands of four: TRACE 1 to 4, DEBUG 5 to 8, INFO 9 to 12, WARN 13 to 16, ERROR 17 to 20 and FATAL 21 to 24. Anything 17 or above is an error. Most applications only need the first number of each band.

The practical rules I hold teams to:

• ERROR means a human should look, eventually. If nobody would act on it, it is a WARN or a metric.
• WARN is for degraded but handled: a retry that succeeded, a fallback taken, a deprecated parameter used.
• INFO is for business events and state changes, one per meaningful thing, not one per function call.
• DEBUG is off in production by default and switchable at runtime per service, ideally per request.

The level is also your first sampling key: 13 and up is rare and valuable, while most of the volume, and the bill, typically lives at 9 and below.

How do you correlate logs with traces?

Put the active trace ID and span ID on every log line as top-level fields. OpenTelemetry's spec for trace context in non-OTLP log formats is Stable and says JSON logs should carry trace_id, span_id and trace_flags as top-level fields in lowercase hex. W3C Trace Context fixes the sizes: 16 bytes (32 hex characters) for a trace ID, 8 bytes (16) for a span ID. With them, a backend can jump from a trace to its logs and back, which is the point of distributed tracing.

Here is the Python setup I use: structlog for JSON, the OpenTelemetry API for the current span, and a small deny-list as a backstop.

# pip install structlog==26.1.0 opentelemetry-api==1.45.1 opentelemetry-sdk==1.45.1
import logging, sys
import structlog
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider

SENSITIVE = {"password", "authorization", "api_key", "token", "cookie", "prompt"}

def add_trace_context(logger, method, event):
    # OTel's JSON convention: top-level trace_id/span_id, lowercase hex
    ctx = trace.get_current_span().get_span_context()
    if ctx.is_valid:
        event["trace_id"] = trace.format_trace_id(ctx.trace_id)
        event["span_id"] = trace.format_span_id(ctx.span_id)
        event["trace_flags"] = f"{ctx.trace_flags:02x}"
    return event

def drop_sensitive(logger, method, event):
    # Deny-list as a backstop; the real control is not passing secrets to the logger
    for key in SENSITIVE & event.keys():
        event[key] = "[REDACTED]"
    return event

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,      # request-scoped fields
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso", utc=True),
        add_trace_context,
        drop_sensitive,
        structlog.processors.format_exc_info,
        structlog.processors.JSONRenderer(),
    ],
    wrapper_class=structlog.make_filtering_bound_logger(logging.INFO),
    logger_factory=structlog.PrintLoggerFactory(file=sys.stdout),
)

trace.set_tracer_provider(TracerProvider())
log = structlog.get_logger(service="checkout")

with trace.get_tracer("demo").start_as_current_span("POST /orders"):
    structlog.contextvars.bind_contextvars(order_id="ord_8812", tenant="acme")
    log.info("payment.authorised", amount_minor=4999, currency="EUR", token="tok_live_abc")
    log.debug("this line is filtered out at INFO")

Running it on Python 3.14.6 printed exactly one line (the DEBUG call is filtered):

{"service": "checkout", "amount_minor": 4999, "currency": "EUR", "token": "[REDACTED]", "event": "payment.authorised", "tenant": "acme", "order_id": "ord_8812", "level": "info", "timestamp": "2026-10-24T10:28:35.574871Z", "trace_id": "8c1e25469fadb5cbe038fb89384bba3b", "span_id": "6ab2bfaa31cb8124", "trace_flags": "03"}

The trace_flags value of 03 surprised me. Bit 0x01 is "sampled" and bit 0x02 is the random-trace-ID flag from W3C Trace Context Level 2, which the Python SDK now sets. Test the bit in queries; do not compare the byte to 01.

Why not use the OpenTelemetry Logs SDK directly? The language status page lists logs as Stable for Java, .NET, C++, PHP and Rust, a release candidate for Go, and still Development for Python and JavaScript. In those two I keep the native library, inject the IDs, and let a collector convert JSON to OTLP; my Collector on Kubernetes guide shows that side.

What not to log: PII, secrets and tokens

OWASP's list of data not to record directly is worth pinning up: session identifiers, access tokens, authentication passwords, database connection strings, encryption keys and other primary secrets, bank account and payment card data, sensitive personal data and some forms of PII, source code, commercially sensitive information, and anything a user has not consented to or that is illegal to collect where you operate.

The failures here are not hypothetical. In May 2018 Twitter urged its users to change their passwords because a bug wrote passwords to an internal log before hashing completed. In September 2024 Ireland's Data Protection Commission fined Meta €91 million for storing some users' passwords in plaintext on internal systems, an issue Meta had reported in 2019. Neither was reported as exposed externally; both were internal systems holding what they should never have seen.

Three controls, in order of how much they help:

1. Allow-list at the source. Log named fields, never request.__dict__, a whole header map or a raw request body. This is the only control that scales.
2. Redact in the pipeline. The OpenTelemetry Collector's redaction processor drops attributes not on an allow-list and masks values matching patterns. It is alpha for logs and works on attributes (and map-shaped bodies), not on plain string bodies, so those need an OTTL replace_pattern too.
3. Deny-list in the logger. Like drop_sensitive above. Never sufficient: the dangerous field is the one nobody thought to name.

Hashing an email does not anonymise it. Under GDPR Recital 26, pseudonymised data that can be attributed to a person with additional information is still personal data. Leaked keys in logs and repositories are their own epidemic, which I covered in Secrets Sprawl 2026.

Log sampling without losing the errors

Sampling logs is fine; sampling them randomly is not. Never sample WARN and above, and sample by trace ID, so that a kept request keeps all its lines and the trace and log samplers agree.

The OpenTelemetry Collector does both. The probabilistic sampler processor uses the trace ID as its randomness source for logs by default and accepts a per-record priority attribute (0 drops, 100 keeps). One default bites: with fail_closed true, a log record with no trace ID counts as an error and is refused, which silently drops startup messages and cron output. This otelcol-contrib v0.163.0 config keeps every warning and error, 10 percent of the rest, and masks obvious tokens:

# otelcol-contrib v0.163.0: keep WARN+ always, sample the rest by trace ID
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  transform/log_hygiene:
    error_mode: ignore
    log_statements:
      # severity_number 13+ is WARN, ERROR or FATAL: never sample these away
      - set(log.attributes["sampling.priority"], 100) where log.severity_number >= SEVERITY_NUMBER_WARN
      # mask bearer tokens and AWS access key IDs that slipped into string bodies
      - replace_pattern(log.body, "Bearer [A-Za-z0-9._~+/-]+=*", "Bearer [REDACTED]")
      - replace_pattern(log.body, "AKIA[0-9A-Z]{16}", "[REDACTED_AWS_KEY]")
  probabilistic_sampler/logs:
    sampling_percentage: 10            # INFO and below: keep 10% of traces
    sampling_priority: sampling.priority
    fail_closed: false                 # default true drops logs without a trace ID
  batch: {}

exporters:
  otlphttp:
    endpoint: https://logs.example.internal

service:
  pipelines:
    logs:
      receivers: [otlp]
      processors: [memory_limiter, transform/log_hygiene, probabilistic_sampler/logs, batch]
      exporters: [otlphttp]

For the noisy-loop problem, a service printing the same connection error in a tight loop, the log deduplication processor collapses identical records over an interval (10 seconds by default) into one with a log_count attribute. It is alpha, but it is the right shape of fix.

How much do logs cost in 2026?

Logs scale with traffic rather than with the number of things you monitor, which is why they are the line on an observability bill that surprises people. Grafana Labs' 2026 Observability Survey of 1,363 respondents found cost was the top tool selection criterion for the third year running, at 65 percent, and that 48 percent report using OpenTelemetry for logs, against 57 percent for metrics and 50 percent for traces.

List prices this month, per gigabyte unless stated:

Backend	Ingest	What else you pay	Source
AWS CloudWatch Logs, Standard class	$0.50/GB (US East)	storage $0.03/GB-month, queries billed separately	AWS pricing
AWS CloudWatch Logs, Infrequent Access	$0.25/GB (US East)	storage $0.03/GB-month	AWS pricing
Grafana Cloud Logs, Pro	$0.05 process + $0.40 write + $0.10 retain	30-day retention, 50 GB included, $19/month platform fee	Grafana pricing
Datadog Log Management	$0.10/GB ingest	indexing $1.70 per million events for 15 days (annual), $2.55 on-demand; Flex storage from $0.05 per million	Datadog pricing

The comparison is not like for like, and that is the lesson. Datadog charges little to ingest and a lot to index, per event rather than per byte, so tiny chatty lines cost more per gigabyte. CloudWatch charges at ingest, then for storage and queries. Grafana bundles 30 days of retention. Before negotiating, know your average event size, your events per second by level, and how many events anyone queried last month.

The cheapest log is the one you never emit: drop DEBUG in production, deduplicate loops, sample INFO by trace, shorten retention on noisy streams, and send compliance copies to object storage instead of an index. In Loki, cost and speed depend on labels: the Loki label best practices say never to extract a trace ID or order ID into a label, and suggest staying under 100,000 active streams per tenant even for tenants sending over 10 TB a day. Put trace IDs in structured metadata or the line itself. The same cardinality arithmetic applies to metrics; see fixing Prometheus high cardinality.

Logging for LLM apps and agents

LLM applications break two assumptions: the payloads are large, and they are user data. A prompt can contain a medical question, a pasted API key, or an injection string aimed at whatever reads your logs later. Treat prompts and completions as user content, not debug output.

OpenTelemetry's GenAI conventions now live in their own semantic-conventions-genai repository, created in May 2026; the old GenAI pages on opentelemetry.io now just point there. Their guidance is explicit: instrumentations should not capture system instructions, inputs or outputs by default, and should offer an opt-in. They describe three patterns: record nothing (the default), record content on spans or events (for pre-production), or upload content to separate storage and record a reference, recommended for production because it allows separate access controls. The content attributes, gen_ai.input.messages, gen_ai.output.messages and gen_ai.system_instructions, are Opt-In with a PII warning, and events such as gen_ai.evaluation.result are still in Development.

What I log for every model call: model and provider, token counts, time to first token and total latency, finish reason, tool names (not arguments), prompt template ID and version, trace and span IDs, and any evaluation score. Content goes to a separate store with shorter retention and stricter access, keyed by response ID. That covers cost and latency debugging, which is most LLM debugging, without copying every conversation into the log index. Today's Langfuse and OpenTelemetry guide covers the tracing side.

The same split applies to agent sandboxes. PandaStack is my company, a Firecracker microVM cloud for agent code execution, and the rule I apply there is that a sandbox's stdout is customer data, not platform logs, so it stays out of the operational pipeline.

FAQ

Should logs be JSON or logfmt?

JSON, for anything a machine will read. Logfmt is pleasant in a terminal, but nesting and escaping are not standardised, and every pipeline parses JSON natively. Render pretty text in development and JSON everywhere else from the same logger config.

What is the difference between a log and an event in OpenTelemetry?

An event is a LogRecord with a non-empty EventName. The name identifies the structure of its body and attributes, so tools can rely on it. Every event is a log record; not every log record is an event.

Should I put the trace ID in Loki labels?

No. Grafana's Loki documentation says not to extract ephemeral values such as trace IDs or order IDs into labels, because each unique value creates a new stream. Keep the trace ID in the log line or in structured metadata and filter on it at query time.

Is it OK to log user IDs and email addresses?

Internal opaque user IDs are usually acceptable and are what makes logs useful in support cases; check your own policy. Email addresses, names and anything OWASP classes as sensitive personal data should be masked or omitted. Hashing alone does not anonymise them under GDPR.

This quarter I would do four things, in order. Pick one JSON schema based on the OpenTelemetry data model and ship it as a shared logging config per language. Add trace_id and span_id to every line and make the trace-to-logs link work in your backend. Write the redaction allow-list and add sampling and dedup in the Collector, with fail_closed: false. Then pull a month of log spend by service and level, because the number that changes behaviour is not the price per gigabyte but how many of your gigabytes are INFO lines nobody has ever read.

Related: Observability vs Monitoring: What's the Difference in 2026?, Distributed Tracing Explained: traceparent, Sampling and Jaeger v2 (2026) and How to Set Up Grafana Loki on Kubernetes with Alloy (2026).