Agent Observability in 2026: The Spans Are Standard, the Standard Isn't Stable, and Nobody Reads the Logs

Every agent that runs on my platform produces two kinds of evidence about what it did. There is the kernel-level record, which I own: what processes it spawned in its microVM, what it wrote, what it connected to. And there is the model-level record, which the customer owns: what the model was asked, what it answered, which tool it decided to call and why. I wrote about the first kind in the 2 AM post. This is about the second kind, which in 2026 finally has a standard shape, and about the uncomfortable evidence from this year's evaluation papers that having the traces is not the same as looking at them.

The standard that everyone ships and nobody has stabilised

OpenTelemetry's GenAI semantic conventions define what a span for a model call or a tool call should be named and what attributes it carries. In 2026 they are the de facto interchange format: Datadog, Honeycomb, Grafana, Sentry, Langfuse, Arize and MLflow all ingest gen_ai.* attributes natively. And yet every single one of those attributes still carries the "Development" badge. None is Stable. The only stable attribute that appears in an agent span is error.type, which OpenTelemetry stabilised years ago for everything else.

The instability is not theoretical. Here is what has changed in the names since the conventions began, from the release notes:

Version	Date	Change
1.27	Aug 2024	prompt_tokens / completion_tokens renamed to input_tokens / output_tokens
1.37	Aug 2025	gen_ai.system renamed to gen_ai.provider.name; per-message events replaced by gen_ai.input.messages and gen_ai.output.messages
1.38	Oct 2025	gen_ai.evaluation.result event added
1.39, 1.40	Jan, Feb 2026	retrieval spans, cache-token attributes, gen_ai.agent.version, MCP conventions
1.41	Apr 2026	invoke_agent split into CLIENT (remote agent) and INTERNAL (in-process) kinds; reasoning tokens; time-to-first-chunk metric
1.42	Jun 2026	everything moved out of the main repo into semantic-conventions-genai

The June move matters practically: as of mid-July the new repository had no tagged release, so there is no versioned schema URL to pin, and the opentelemetry.io pages for GenAI now say "moved, no longer maintained here." If you built dashboards against the 1.37 names in 2025, the 1.41 split of invoke_agent into two span kinds probably broke your agent-latency panel this spring. Sentry follows 1.36, Honeycomb 1.40, and the Python instrumentation for OpenAI emits the frozen 1.30-era attributes unless you set OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental. Three vendors, three vintages of the same "standard," and a customer whose agent uses all three sees three shapes of span.

I do not say this to be dismissive. The conventions are good and the SIG is doing careful work; the compliance matrix in the new repo tracks about 24 Python libraries from the Anthropic and OpenAI SDKs through LangChain, Pydantic AI, CrewAI and the Claude Agent SDK. The point is that "OTel-native" on a vendor's marketing page tells you nothing until you know which version they parse, and you should keep that number somewhere you can find it.

What the spans actually look like

I wanted to see the emitted spans rather than read the spec, so I installed the official Python GenAI utilities into a fresh virtualenv this morning and wrote a fake agent loop: one invoke_agent span, a stub model that asks for a read_file tool on the first turn and answers on the second, and a tool span between them. The library is opentelemetry-util-genai 1.1b0, which ships a TelemetryHandler with context managers for each span type, on top of opentelemetry-semantic-conventions 0.65b0, whose gen_ai_attributes module now defines 60 attribute names.

from opentelemetry.util.genai.handler import get_telemetry_handler
from opentelemetry.util.genai.types import (InputMessage, OutputMessage, Text,
                                           ToolCallRequest, ToolCallResponse)
h = get_telemetry_handler()

with h.invoke_local_agent(agent_name="hostname-bot", request_model="stub-1") as agent:
    agent.conversation_id = "conv-42"
    msgs = [InputMessage(role="user", parts=[Text(content="What's the hostname?")])]
    for step in range(5):                                  # hard step budget
        with h.inference("stub", request_model="stub-1") as llm:
            llm.input_messages = list(msgs)
            kind, part = fake_model(msgs)                  # returns a ToolCallRequest or Text
            llm.input_tokens, llm.output_tokens = 120 + 30 * step, 18
            llm.output_messages = [OutputMessage(role="assistant", parts=[part],
                                   finish_reason="tool_call" if kind == "tool" else "stop")]
        if kind == "text":
            agent.output_messages = llm.output_messages
            break
        with h.tool(part.name, tool_call_id=part.id, tool_type="function") as t:
            t.arguments = part.arguments
            t.tool_result = read_file(**part.arguments)
        msgs += [InputMessage(role="assistant", parts=[part]),
                 InputMessage(role="tool", parts=[ToolCallResponse(response=t.tool_result, id=part.id)])]

With a console exporter and no content-capture setting, this is the complete output, four spans:

chat stub-1          CLIENT   gen_ai.operation.name=chat gen_ai.request.model=stub-1
                              gen_ai.provider.name=stub gen_ai.response.finish_reasons=[tool_call]
                              gen_ai.usage.input_tokens=120 gen_ai.usage.output_tokens=18     21.1 ms
execute_tool read_file INTERNAL gen_ai.operation.name=execute_tool gen_ai.tool.name=read_file
                              gen_ai.tool.call.id=call_1 gen_ai.tool.type=function              6.3 ms
chat stub-1          CLIENT   … finish_reasons=[stop] input_tokens=150 output_tokens=18       20.2 ms
invoke_agent hostname-bot INTERNAL gen_ai.operation.name=invoke_agent gen_ai.agent.name=hostname-bot
                              gen_ai.conversation.id=conv-42                                   47.9 ms

No prompt text, no tool arguments, no tool result. That is the default and it is the right default. Setting OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY and re-running adds gen_ai.input.messages and gen_ai.output.messages to every chat span as JSON strings, gen_ai.tool.call.arguments and gen_ai.tool.call.result to the tool span, and the final answer to the agent span. The other modes are EVENT_ONLY, which sends content as a gen_ai.client.inference.operation.details log event instead of a span attribute, and SPAN_AND_EVENT. The distinction is the whole PII design: spans go to your tracing backend where everyone with dashboard access can see them; events can be routed by the Collector to a separate, access-controlled log store, or dropped.

For a real model the instrumentation packages do the chat span for you. The names are a trap, so here they are: opentelemetry-instrumentation-openai-v2 (2.4b0, May) is the official OpenTelemetry one; opentelemetry-instrumentation-genai-anthropic (1.1b1, 21 August) is the official Anthropic one from the new opentelemetry-python-genai repo; and opentelemetry-instrumentation-anthropic on PyPI, despite the name, is Traceloop's OpenLLMetry package with its own content flag, TRACELOOP_TRACE_CONTENT. The agent frameworks have their own layers: the OpenAI Agents SDK traces by default to OpenAI's dashboard with no OTLP export unless you add a processor; Google ADK 1.17 has OpenTelemetry built in and its sample config sets EVENT_ONLY content capture; Vercel's AI SDK 7 registers the GenAI span names through @ai-sdk/otel; and Claude Code exports metrics and events with CLAUDE_CODE_ENABLE_TELEMETRY=1, redacting prompts and tool content unless you separately opt in, and passes TRACEPARENT to its subprocesses so a trace continues through the shell commands it runs.

That last idea is now in the protocol. The MCP revision of 28 July, which I covered from the security side in the MCP post, adopted SEP-414: W3C traceparent, tracestate and baggage keys in the _meta field of requests, so a host's trace continues through the MCP client into the server and whatever it calls. The same revision deprecated MCP's own logging capability in favour of stderr for stdio servers and OpenTelemetry for everything structured. The OTel side has a matching mcp.md convention with spans named tools/call get-weather and metrics for operation and session duration. For the first time an agent, its tools and the servers behind those tools can share one trace ID without anyone writing glue.

The vendors

The money followed the standard. Langfuse, the most-starred open-source option at 34,000 stars under MIT, was acquired by ClickHouse on 16 January, the day ClickHouse announced a $400 million round; it stays open source and self-hostable, and it ingests OTLP at a single endpoint, mapping gen_ai.*, OpenInference and MLflow attributes. Braintrust raised $80 million in February at a reported $800 million valuation. LangChain raised $125 million at $1.25 billion in October for LangSmith. Promptfoo, the eval tool, joined OpenAI in March and stays MIT. Humanloop's team went to Anthropic and the product is being shut down. Weights & Biases went to CoreWeave last year. Gartner published its first Market Guide for the category in February and predicts 60% of engineering teams will use one of these platforms by 2028, from 18% in 2025.

The general-purpose observability vendors arrived in a cluster: Datadog's agent monitoring went GA in June 2025, Honeycomb launched Agent Observability on 12 May with an agent timeline view, and Grafana Cloud's went GA on 30 July. Arize's Phoenix, at 11,000 stars under the Elastic licence, uses its own OpenInference conventions with translation. MLflow 3.16 shipped on 3 September with span links. If you already pay one of these, the answer to "which agent observability tool" is probably "the one you already pay," because the ingest format is shared. The specialist tools differentiate on evaluation, which is where the real problem is.

What actually catches failures

The evaluation literature this year has one consistent message, and it is that the aggregate numbers we report about agents are less informative than we think.

Pass@k lies about reliability. τ-bench introduced pass^k, the probability that all k attempts succeed, alongside pass@k, at least one succeeding. In their own tables Claude 3.5 Sonnet on the retail domain went from 69% at pass^1 to 46% at pass^4, and on airline from 46% to 22%. A product that runs the agent once per customer experiences pass^1; a customer who uses it four times experiences pass^4. Anthropic's evals guide from January recommends reporting both. A March paper on 23,000 episodes across ten models proposes reliability-decay curves as the primary metric.

The benchmarks are saturated or contaminated. OpenAI stopped reporting SWE-bench Verified on 23 February, after finding that about 59% of the 138 hardest tasks had flawed tests and that models could reproduce patches from the task ID alone; top scores had moved only from 74.9% to 80.9% in six months. The replacement, SWE-bench Pro, has held-out and commercial splits precisely so that cannot recur. Terminal-Bench 2.0, released in November, is already superseded by 4.0 with incomparable scores. When a vendor quotes a number to you, ask which version.

Nobody reads the logs, and it matters. Princeton's Holistic Agent Leaderboard paper, 21,730 rollouts across nine models and nine benchmarks at a cost of about $40,000, found by inspecting logs that agents were searching Hugging Face for the benchmark answers and misusing test credit cards, and that higher reasoning effort reduced accuracy in most runs. The follow-up in May, Log analysis is necessary for credible evaluation of AI agents, found that on τ-bench's airline domain the pass^5 score had been under-elicited by roughly half, because harness bugs visible in the traces were being scored as model failures. The 2025 MAST taxonomy of multi-agent failures got its 14 failure modes the same way, by human annotation of 150 traces with an inter-rater agreement of 0.88; the largest categories were system design and inter-agent misalignment, not model capability.

The pattern is that trace collection is solved and trace reading is the bottleneck. Every finding above came from a human, or an LLM annotator validated against humans, looking at complete trajectories. That is what the content-capture modes exist for, and it is why "we have observability" is not the same as "we know how our agent fails."

What I do with this

Four things, none clever.

A step budget on every loop. My stub above stops at five turns. The reason is not elegance; it is that the runaway-loop stories, the agent that re-reads the same file forty times or compacts its context in a cycle, are the failures that cost money at 2 AM, and a max_steps with error.type=max_steps_exceeded on the agent span is the only control that works when the model is confidently wrong. I could not find a primary source for any of the viral "$47,000 loop" anecdotes, so I will not cite them, but I have my own smaller ones.

Tail-sample on cost and failure, head-sample the rest. Agent traces are big: one invoke_agent can hold dozens of chat spans each carrying the full message history if content capture is on. Langfuse samples at trace level so a kept trace keeps all its observations. In the Collector, keep 100% of traces with an error.type, with latency over the SLO, or with token usage over a threshold, and 5% of the rest. The expensive traces are the interesting ones by construction.

Content off by default, on by route. NO_CONTENT in production spans. EVENT_ONLY where you need trajectories for eval, with the Collector's redaction and transform processors routing those events to a store with its own access control and retention. Never gen_ai.tool.call.result on a span for a tool that reads files, because the file will be a .env eventually.

An eval harness that reports pass^k and stores trajectories. Twenty lines:

import statistics
def evaluate(cases, run, grade, k=4):
    rows = []
    for c in cases:
        outcomes = [grade(c, run(c["input"])) for _ in range(k)]   # grade: final state + required tool calls -> bool
        rows.append({"id": c["id"], "pass@1": sum(outcomes) / k,
                     "pass@k": any(outcomes), "pass^k": all(outcomes)})
    return {m: statistics.mean(r[m] for r in rows) for m in ("pass@1", "pass@k", "pass^k")}

The grader should check end state, a database diff or a file tree, and the set of tool calls that had to happen, not the final text. Each score goes out as a gen_ai.evaluation.result event keyed by the gen_ai.response.id it judged, so the eval and the trace it evaluated live in the same system and a regression is one query away from its trajectory. If an LLM is the grader, swap the argument order and average, because position bias in judges is well documented, and never let the judge be the model under test.

None of this requires a stable standard. It requires deciding that the trajectories are the product's most important log, and reading some of them every week. The conventions will stabilise eventually; the habit is the part nobody can ship to you.

---

Related: It's 2 AM. Do You Know What Your AI Agent Is Doing?, MCP Security in 2026 and Who Is This Agent? Non-Human Identity in 2026.