LLM Observability Tools Compared (2026): Langfuse vs LangSmith vs Phoenix

Choosing an LLM observability tool in 2026 looks like a feature comparison and turns out to be a billing and data-gravity decision. Every product on the shortlist draws a trace tree and shows token counts. Where they differ is what they count on the invoice, where your prompts end up living, and how painful it is to leave once your dashboards depend on their attribute names. I have spent fourteen years deciding where telemetry goes, and those questions have not changed. This post compares Langfuse, LangSmith, Arize Phoenix, Helicone, Braintrust, Datadog and OpenLLMetry on what to trace, the state of the OpenTelemetry GenAI conventions, what one agent workload costs at list price, and self-hosting. I checked prices on 31 October 2026; version snapshots are from 24 October unless noted. The hands-on setup is in the companion guide on tracing an LLM app with OpenTelemetry and Langfuse.

What should you trace in an LLM app?

Five things, and the attribute names matter as much as the data, because they decide which backends can read your history later.

Tokens, split by kind. Input and output are the obvious pair. The current GenAI conventions also define gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_write.input_tokens and gen_ai.usage.reasoning.output_tokens on the inference span, per the semantic-conventions-genai repository. Cached and reasoning tokens are priced differently, so a single "total tokens" number stops meaning anything the day you turn on prompt caching.

Cost. There is no cost attribute in the OpenTelemetry GenAI registry, so every tool computes cost itself by matching the model name against a price table. Langfuse's cost tracking docs are candid about the consequences: inferred cost is calculated at ingestion and never recalculated when a price changes, and overlapping usage buckets double-count. Check how your tool prices a model it has never heard of, such as your own vLLM deployment.

Latency, including time to first token. For streaming the conventions define gen_ai.response.time_to_first_chunk, the number users actually feel and the one that belongs on the SLO, as I argued in SRE for LLM apps.

Tool calls as their own spans. The spec's execute_tool {gen_ai.tool.name} span, with gen_ai.tool.call.id linking it to the model's request, is where agents go wrong: bad arguments, timeouts, a tool that pours 40 KB of JSON into the context. Arguments and results are opt-in because they often contain customer data. I build PandaStack (my own company, a Firecracker microVM cloud for agents), where a tool call is often code running in a sandbox, so this is the span I care about most.

Sessions, users and scores. A conversation is many traces. You need a session ID (gen_ai.conversation.id in the spec) and a user ID to answer "what did this user's agent do yesterday", and scores from feedback or evaluators on the trace, so quality can be filtered next to cost.

Check one default before production: content capture. The official OpenTelemetry instrumentation for OpenAI records no prompt text unless you set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, per its README, while Langfuse's SDK captures inputs and outputs by default. Know which you have before a customer's home address lands in someone else's database.

Are the OpenTelemetry GenAI semantic conventions stable yet?

No. Everyone ingests them, nobody has stabilised them, and they are still changing month to month.

The GenAI conventions now live in their own repository, created in May 2026. On 24 October it has no tagged release, the README's schema URL section still reads "TODO", and every one of the 79 gen_ai.* attributes in its registry carries the Development badge. The main semantic-conventions repository's latest release is v1.44.0, from 4 August.

Development status allows breaking changes, and the repo's pending changelog holds ten. Two hit dashboards directly. On 22 September the gen_ai.client.token.usage histogram, the metric most token dashboards were built on, was replaced by separate per-direction usage metrics broken down by modality, cache and reasoning. On 6 October inference got its own gen_ai.client.inference.duration metric and the time-to-first-chunk metrics were renamed into the gen_ai.client.inference.* namespace.

The instrumentation moved too: the official Python package for OpenAI is now opentelemetry-instrumentation-genai-openai (1.3b0, 23 October), successor to opentelemetry-instrumentation-openai-v2, and it always emits the latest experimental conventions with no opt-in switch.

Vendors cope by pinning or translating. Datadog's OTLP intake accepts conventions from 1.37 onward and drops spans with no gen_ai.* attribute. Phoenix has converted gen_ai.* into its OpenInference schema since 15.10.0. LangSmith's OTel endpoint and Langfuse's OTLP endpoint each map gen_ai.* plus two or three other dialects. The rule: emit OpenTelemetry, record which convention version you target, and expect to fix a dashboard after each upgrade. I covered the span shapes in Agent Observability in 2026.

Langfuse, LangSmith, Phoenix and the rest: one paragraph each

Langfuse is the default open-source choice: about 36,800 GitHub stars, MIT-licensed outside its ee/ directories, and owned by ClickHouse since a January 2026 acquisition that promised no licence changes. Server v4 shipped on 29 July with full-text search, monitors and faster APIs, and v4.72.0 (29 October) is current. Its SDK is itself an OpenTelemetry tracer.

LangSmith is LangChain's platform and the easy path on LangGraph, though it accepts OTel from anything. It is closed source, with self-hosted or hybrid deployment on Enterprise. Read the retention rules twice: base traces keep 14 days and extended traces 180 days, a cap that applies to new SaaS traces from 14 September per the administration docs, and online evaluators upgrade the traces they score to extended retention unless you opt out.

Arize Phoenix is the open-source, local-first half of Arize: one container image, OpenInference-native, Elastic License 2.0, version 21.0.0 since 30 October. ELv2 lets you run it yourself with no fees or feature gates but forbids offering it to third parties as a hosted service. Arize AX is the commercial SaaS.

Helicone took the proxy route: point your OpenAI base URL at it and every request is logged, with routing and fallbacks on top. It is Apache-2.0. In March 2026 Helicone joined Mintlify and the product went into maintenance mode, with security fixes and new model support but no new features (that write-up is from a competitor). Today its pricing page carries a banner saying it is no longer part of Mintlify, details to come. I would not start a new project on it.

Braintrust is eval-first: experiments, scorers and datasets fed by production logs, metered on gigabytes and scores. Self-hosting is an Enterprise-only hybrid: the data plane runs in your cloud, the UI and auth stay with Braintrust.

Datadog sells LLM Observability, now listed as Agent Observability on its pricing page, as a standalone product that bills only LLM spans, not tool, retrieval or workflow spans. Correlation with your existing Datadog APM is the selling point. Retention is 15 days by default, with longer retention sold as an extra, and there is no self-hosted option.

OpenLLMetry is not a backend. It is Traceloop's Apache-2.0 set of OpenTelemetry instrumentations, version 0.62.4, covering 16 model providers, seven vector databases, ten frameworks and MCP, with 24 tested destinations listed in its README. Traceloop announced in March 2026 that it was joining ServiceNow, and says OpenLLMetry stays open source.

LLM observability pricing compared: what does one agent workload cost?

No two vendors meter the same thing, so fix a unit of work: one agent run producing a trace with an agent span, four LLM calls, three tool calls and one score. Count what each vendor bills for it, then price 300,000 runs a month.

Langfuse counts every trace, observation and score, per its billable units page, so a run is 10 units and the month is 3 million. Core is $29 with 100,000 units included, then $8 per 100,000 up to a million and $7 per 100,000 from one to ten million: $29 + $72 + $140 = $241. Pro, with the same metering and three years of data access, comes to $199 + $212 = $411. Datadog bills only the four LLM spans, 1.2 million a month. Its Pro tier includes 100,000 spans for $160 a month on an annual plan, and the other 1.1 million at $3.50 per 10,000 add $385, so $545. Month-to-month it is $200 + $462 = $662, and on-demand $240 + $550 = $790.

For the other four I could not finish the sum from public pages. LangSmith says only "pay as you go" past its included traces, Arize AX and Helicone list no overage rate, and Braintrust bills on bytes, so the answer depends on your prompt sizes. If a vendor will not tell you the overage price before you sign, assume it is high.

In the same runs, free tiers cover 10,000 a month on Datadog, 5,000 on Langfuse and LangSmith, 3,125 on Arize AX and 2,500 on Helicone.

Tool	Licence and self-hosting	Free tier	Paid entry	OpenTelemetry ingest
Langfuse	MIT core; free self-host on Compose or Helm	Hobby: 50k units, 30 days, 2 users	Core $29/mo with 100k units, then $8 per 100k	OTLP over HTTP; maps gen_ai.*, OpenInference, MLflow
LangSmith	Proprietary; self-host or hybrid on Enterprise	Developer: 5k base traces, 1 seat	Plus $39 per seat, 10k base traces	OTLP; maps gen_ai., OpenInference, traceloop.
Arize Phoenix / AX	Phoenix ELv2, free self-host; AX self-host on Enterprise	AX Free: 25k spans, 1 GB, 15 days	AX Pro $50/mo, 50k spans, 10 GB, 30 days	OpenInference native; converts gen_ai.* since 15.10.0
Helicone	Apache-2.0; maintenance mode since March	Hobby: 10k requests, 7 days	Pro $79/mo, 10k requests, 1 month	Proxy (base URL swap)
Braintrust	Proprietary; hybrid data plane on Enterprise	Starter: 1 GB, 10k scores, 14 days	Pro $249/mo, 5 GB, 50k scores, 30 days	OTLP at /otel/v1/traces
Datadog	Proprietary, SaaS only	40k LLM spans, 15 days	Pro $160/mo annual, 100k LLM spans, then $3.50 per 10k	OTLP, GenAI 1.37+, dd-otlp-source=llmobs
OpenLLMetry	Apache-2.0 instrumentation, no backend	n/a	n/a	Emits OTel to 24 tested destinations

Can you self-host an LLM observability tool?

Langfuse, Phoenix and Helicone run on your own hardware without a contract; LangSmith, Braintrust and Arize AX offer it on Enterprise; Datadog does not. Between Langfuse and Phoenix the difference is operational weight. Langfuse v4 is a web container and a worker on top of Postgres, ClickHouse, Redis and S3-compatible storage. Its Docker Compose docs ask for at least 4 cores and 16 GiB of memory and are explicit that Compose gives you no high availability, scaling or backups; that is what the Helm chart is for. Batches land in object storage first and the worker loads ClickHouse from there, per the architecture overview, so a database outage loses nothing. Sound design, and also five stateful components to patch and back up. Phoenix ships as one container image and advertises air-gapped operation.

Self-hosting is a platform-engineering decision wearing an AI badge. The telemetry holds your customers' messages, which is the reason to keep it in your VPC and also the reason it needs the backup, retention and access control of a primary database. Langfuse's free edition includes organisation-level RBAC and enterprise SSO, but project-level RBAC, data retention management and audit logs are in the paid self-hosted Enterprise edition. If you need data deleted after N days for GDPR, either budget for that edition or plan how you will delete old traces yourself.

How to instrument once and switch LLM observability backends

The cheapest insurance is to keep vendor code out of your application: instrument with OpenTelemetry and pick the backend with environment variables. Endpoints and headers come from the vendor docs linked above.

# Python 3.11+. Versions current on 24 October 2026.
pip install "openai==3.37.0" \
  "opentelemetry-sdk==1.45.1" \
  "opentelemetry-exporter-otlp-proto-http==1.45.1" \
  "opentelemetry-instrumentation-genai-openai==1.3b0"

export OTEL_SERVICE_NAME=support-agent
export OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
# Prompt and completion text on spans; off by default, think before enabling
export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=span_only

# Pick ONE backend block.
# Langfuse Cloud EU (self-hosted: http://localhost:3000/api/public/otel)
export OTEL_EXPORTER_OTLP_ENDPOINT="https://cloud.langfuse.com/api/public/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic $(printf '%s' "$LF_PK:$LF_SK" | base64 | tr -d '\n'),x-langfuse-ingestion-version=4"

# Phoenix, self-hosted on its default port
# export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="http://localhost:6006/v1/traces"

# LangSmith
# export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.smith.langchain.com/otel"
# export OTEL_EXPORTER_OTLP_HEADERS="x-api-key=$LANGSMITH_API_KEY,Langsmith-Project=support-agent"

# Braintrust
# export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.braintrust.dev/otel"
# export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer $BRAINTRUST_API_KEY,x-bt-parent=project_id:$BT_PROJECT_ID"

# Datadog: use your Datadog site's OTLP traces intake URL
# export OTEL_EXPORTER_OTLP_TRACES_HEADERS="dd-api-key=$DD_API_KEY,dd-otlp-source=llmobs"
# export OTEL_RESOURCE_ATTRIBUTES="dd_llmobs_enabled=true"

# trace_any_backend.py: the application code never names a vendor
from openai import OpenAI
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.genai.openai import OpenAIInstrumentor
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()  # service.name comes from OTEL_SERVICE_NAME
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))  # reads OTEL_EXPORTER_OTLP_*
trace.set_tracer_provider(provider)
OpenAIInstrumentor().instrument()  # chat, Responses API and embeddings become gen_ai.* spans

client = OpenAI()
resp = client.chat.completions.create(
    model="gpt-5-mini",
    messages=[{"role": "user", "content": "One sentence on time to first token."}],
)
print(resp.choices[0].message.content)
provider.shutdown()  # flush buffered spans before the process exits

Keep the x-langfuse-ingestion-version=4 header: without it, Langfuse says OTel data can take up to ten minutes to appear. If model traffic already goes through a gateway such as LiteLLM, it can emit the same spans at the choke point, as I covered in AI Gateways in 2026.

Which LLM observability tool should you choose?

• You already run Datadog APM and SaaS is fine. Use Datadog. Billing only LLM spans is the most predictable meter for tool-heavy agents. Check that the default 15 days of retention satisfies audit, or price the extension.
• Prompts must stay in your VPC and you want no contract. Langfuse, if you can operate ClickHouse and friends; Phoenix, if you want one container and will never resell it as a service.
• You build on LangGraph. LangSmith, with the overage price in writing.
• Evaluation is the main job. Braintrust, budgeted by bytes, or the experiments in Langfuse and Phoenix.
• You are on Helicone today. Plan the migration on your schedule rather than its.

Whatever you choose, instrument with OpenTelemetry (official GenAI packages or OpenLLMetry) and put unit counts in the same review as the token bill, the discipline I described in FinOps for AI.

FAQ

What is the best open-source LLM observability tool in 2026?

Langfuse is the most complete: MIT-licensed core, tracing, prompts, datasets and evaluations in one product, self-hosted free on Docker Compose or Kubernetes. Arize Phoenix is lighter as a single container, under the Elastic License 2.0, which forbids offering it as a hosted service. OpenLLMetry is instrumentation, not a backend.

Is Langfuse free to self-host?

Yes. The self-hosted open-source edition is MIT-licensed with no usage limits and includes organisation-level RBAC and enterprise SSO. Project-level RBAC, data retention management, audit logs and SCIM are paid Enterprise features. Your real cost is the infrastructure: Postgres, ClickHouse, Redis and object storage on at least 4 cores and 16 GiB.

Does Datadog charge for tool call spans?

No. Datadog's Agent Observability bills only LLM spans, meaning calls to a model provider; tool, retrieval and workflow spans are free. The free tier covers 40,000 LLM spans a month, and Pro starts at $160 a month (annual) for 100,000, then $3.50 per 10,000.

Are the OpenTelemetry GenAI semantic conventions stable?

No. As of 24 October 2026 all 79 gen_ai.* attributes are marked Development, the dedicated repository has no tagged release, and ten breaking changes are pending, including the September replacement of the gen_ai.client.token.usage metric. Use them anyway, but record the convention version your instrumentation targets.

This quarter I would instrument every new agent with the OpenTelemetry GenAI packages rather than a vendor SDK, write the convention version into the repository next to the dashboards, and run Langfuse self-hosted for anything carrying customer prompts, keeping Datadog only where APM correlation already pays for itself. I would also add one line to the monthly cost review: billable units per agent run. The observability invoice scales with agent steps, not users, so it grows every time an agent gets another tool.

Related: Agent Observability in 2026: The Spans Are Standard, the Standard Isn't Stable, and Nobody Reads the Logs, SRE for LLM Apps in 2026: SLOs for TTFT, Provider Outages and Evals and AI Gateways in 2026: Six Options Compared After LiteLLM's 40-Minute Backdoor.