LLM Structured Outputs Explained (2026): JSON Mode vs Strict Schemas

Every LLM integration I have reviewed in the last two years has a function somewhere called something like clean_json(). It strips markdown fences, trims a trailing comma, and occasionally regexes out a stray "Sure! Here is your JSON:". LLM structured outputs exist so you can delete that function. In 2026 all three big API providers ship a mode that compiles your JSON Schema into a grammar and stops the model from emitting a token that would break it. This post explains what JSON mode, structured outputs and strict tool calling each guarantee, how constrained decoding works underneath, which schema features each provider rejects, and the failure modes that survive even a perfect grammar. If you want the code first, the sibling post is How to Get Reliable JSON From LLMs With Pydantic.

JSON mode vs structured outputs vs tool calling: what is the difference?

These names get used interchangeably. They are not the same guarantee.

JSON mode promises syntactically valid JSON and nothing more. OpenAI still offers it as {"type": "json_object"}, and its structured outputs guide is blunt: JSON mode does not enforce schema adherence, your prompt must mention JSON somewhere or the API errors, and without that instruction the model "may generate an unending stream of whitespace". You get an object, with whatever keys the model felt like.

Structured outputs (OpenAI's text.format with type: "json_schema", Anthropic's output_config.format, Gemini's response_format with a schema) promise that the response conforms to the schema you sent, within each provider's supported subset of JSON Schema. This is the mode you want for extraction, classification and anything a downstream parser reads.

Strict tool calling is the same machinery applied to tool arguments instead of the final message. OpenAI's function calling guide says strict mode is built on its structured outputs feature; Anthropic's strict: true on a tool uses the same grammar pipeline as its JSON outputs, according to its structured outputs docs. Use tool calling when the model is choosing an action, and the response format when you simply want data back. One subtlety worth knowing: OpenAI's Responses API tries to normalise tool schemas into strict mode when you omit strict, while Chat Completions "remain non-strict by default". The same tool definition can behave differently depending on which endpoint you call. If you are building agents on top of this, my tool calling guide covers the loop around it.

How does constrained decoding work?

A language model produces a probability distribution over its whole vocabulary at every step. Constrained decoding inserts one more step before sampling: compute which tokens are legal given the grammar and the text so far, set every other token's probability to zero, then sample. If the schema says the next thing must be "currency": followed by one of three strings, the tokens for "cur are allowed and the token for Sure! is not.

The hard part is speed. Tokenizers have over 100,000 entries, a nested schema needs a pushdown automaton rather than a regex, and the mask must be ready before the next forward pass finishes. Three open-source engines dominate:

• XGrammar from the MLC team. Its paper claims "up to 100x speedup over existing solutions", and the project says it is the default structured generation backend in vLLM, SGLang, TensorRT-LLM and MLC-LLM.
• llguidance, the Rust core behind Microsoft's Guidance. Its README reports around 50 microseconds of CPU time per token mask for a 128k-token tokenizer and lists 2025-05-20 as the date it shipped in OpenAI for JSON Schema. It is also merged into vLLM, SGLang, llama.cpp and Chromium.
• Outlines from .txt, the library that popularised the finite-state-machine approach. Version 1.3.3 shipped in August 2026.

The practical consequence for API users is the first-request penalty, mainly on Anthropic. OpenAI's guide now says only fine-tuned models pay extra latency on the first request with a new schema; other models do not. Anthropic compiles your schema into a grammar on first use and documents the cache as 24 hours from last use, a 180-second compile timeout, and a 400 error ("Schema is too complex for compilation") when the grammar is too big. Changing a field's description does not invalidate the compiled grammar; changing the structure does. Changing output_config.format also invalidates the prompt cache for that conversation, which matters if you rely on prompt caching for cost.

Self-hosting gives you the same machinery with a backend switch. With vLLM's OpenAI-compatible server the request looks like this, per the vLLM structured outputs docs:

# pip install "vllm==0.32.0" "openai==3.42.0" "pydantic==2.14.2"
# Server: vllm serve Qwen/Qwen3-8B --structured-outputs-config.backend xgrammar
#   (omit the flag, or pass auto, to let vLLM pick a backend per request)
from typing import Literal
from openai import OpenAI
from pydantic import BaseModel

class Ticket(BaseModel):
    severity: Literal["sev1", "sev2", "sev3"]
    service: str
    summary: str

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")

resp = client.chat.completions.create(
    model="Qwen/Qwen3-8B",
    messages=[{"role": "user", "content": "Classify: checkout returns 502 for all EU users"}],
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "ticket", "schema": Ticket.model_json_schema()},
    },
)
ticket = Ticket.model_validate_json(resp.choices[0].message.content)  # still validate

Note that the old guided_json request field was removed in vLLM 0.12.0; code that still sends it needs to move to structured_outputs. Serving details are in deploying vLLM on Kubernetes.

OpenAI vs Anthropic vs Gemini: structured outputs compared

The feature names converged. The supported schema subsets did not. This is what each provider's current docs say:

	OpenAI	Anthropic (Claude)	Google Gemini
Request field	text.format json_schema (Responses); strict: true on functions	output_config.format json_schema; strict: true on tools	response_format with mime_type: application/json and schema
Python helper	client.responses.parse(text_format=Model)	client.messages.parse(output_format=Model)	pass Model.model_json_schema(), then model_validate_json
Status	Supported since gpt-4o-mini and gpt-4o-2024-08-06	Stable; the old output_format field needs beta header structured-outputs-2025-11-13	Documented, subset of JSON Schema
Published limits	5,000 properties, 10 nesting levels, 1,000 enum values, 120,000 chars of names	20 strict tools, 24 optional params, 16 union-typed params per request	Complex schemas rejected with 400 INVALID_ARGUMENT
Number and string constraints	minimum, maximum, pattern, format supported	Numeric and length constraints unsupported (SDK moves them into descriptions)	minimum/maximum enforced; minLength/maxLength silently ignored
Recursive schemas	Supported	Not supported	Example uses $ref: "#"
Optional fields	All fields required; use a null union	Required first, then optional, in schema order	null in a type array
Source	OpenAI guide	Anthropic docs	Gemini docs

Two rows deserve emphasis. First, "optional" means different things. OpenAI requires every property to be listed in required and expects you to model optionality as ["string", "null"]; Anthropic accepts genuinely optional properties but caps how many you can have across a request. Second, the constraint rows are where silent bugs live. Gemini strips minLength and uniqueItems without telling you. Anthropic's SDK helpers remove unsupported constraints and append them to the field description as a hint, then validate the response against your original schema. I ran anthropic.transform_schema() from the 2.3.0 Python SDK on a Pydantic model with Field(ge=1, le=1000) and the resulting schema contained no minimum at all, just a description reading {maximum: 1000, minimum: 1}. The grammar cannot enforce that. Your validator has to.

What JSON Schema features are actually supported?

Treat each provider's schema support as a dialect, not as JSON Schema. Across all three, the safe core is: objects with additionalProperties: false, the basic types, enum of strings, arrays of objects, anyOf for nullable fields, and internal $defs references. Everything beyond that needs checking against the specific docs:

• Composition (allOf, not, if/then/else) is unsupported on OpenAI. Anthropic supports allOf except combined with $ref.
• Regex works on OpenAI and Anthropic, but Anthropic excludes backreferences, lookaround and word boundaries. Note that the 2.3.0 Python SDK's transform_schema() still moved my pattern into the description, so check what your SDK actually sends.
• Root type must be an object on OpenAI, and it cannot be an anyOf. A top-level list needs wrapping in {"items": [...]}.
• Enum casing is a quirk Anthropic documents itself: Claude may change the capitalisation of enum string values, so compare case-insensitively and never define two values that differ only in case.

This is the best argument for generating schemas from Pydantic or Zod and letting the provider SDK transform them. The SDKs know their dialect, which changes between releases.

How reliable are structured outputs, really?

The best independent data is still JSONSchemaBench (Geng et al., revised February 2025), which ran 10,000 real-world schemas through Guidance, Outlines, llama.cpp, XGrammar, OpenAI and Gemini. Its headline metric is "empirical coverage": the share of schemas for which an engine both accepted the schema and produced output that validated.

Read that chart carefully. The API providers did not produce broken JSON; they refused schemas that used features outside their subset. The paper says OpenAI hit 1.00 compliance on most datasets it covered and attributes this to a conservative strategy that implements only part of JSON Schema. The open engines accepted more schemas and occasionally emitted output that failed validation, or timed out (Outlines' compliance dropped to 0.06 on the hardest set because of generation timeouts). On the hardest dataset, GitHub Hard, the best engine covered 0.41 of schemas.

So "structured outputs are 100% reliable" is true in a narrow sense: if the provider accepts your schema, the bytes conform. The data is from early 2025 and the API subsets have widened since, but nobody has republished the benchmark against 2026 models. If someone has, I could not find it.

Does forcing JSON output make LLMs worse at reasoning?

This was the big argument of 2024. Let Me Speak Freely? (Tam et al., 2024) reported "a significant decline in LLMs reasoning abilities under format restrictions", with stricter formats hurting more. The .txt team, who build Outlines, published a rebuttal titled "Say What You Mean" arguing that the paper's structured and unstructured prompts were not like-for-like and that it conflated JSON mode with constrained generation.

JSONSchemaBench then measured it directly and found the opposite on its tasks. On GSM8K, the unconstrained model scored 80.1% and every constrained engine scored higher, with Guidance at 83.8% and XGrammar at 83.7%. The gains are small and came from one 8B Llama model, so do not generalise them.

My reading of the evidence: constraints themselves do not make models dumber, but a schema that forces the answer before the reasoning can. If the first property is "verdict", the model commits to a verdict in its first few tokens. Put a short "rationale" field before the answer field, since both OpenAI and Anthropic generate keys in schema order. Anthropic adds a caveat: a property asking for detailed step-by-step reasoning can trigger a reasoning_extraction refusal, so ask for a brief explanation, and use the model's native thinking for the heavy lifting. Its docs say grammars do not apply inside thinking blocks, so the model can think freely and still emit conforming JSON at the end. Measure it with your own evals; running LLM evals in CI is how I would catch a regression from a schema change.

When structured outputs still fail

A grammar guarantees shape. It does not guarantee a complete response, a truthful one, or a response at all. These are the failures I design for:

1. Refusals. Anthropic returns HTTP 200 with stop_reason: "refusal", bills you, and the output may not match your schema. OpenAI returns a refusal content item instead of your object. Code that blindly parses will crash on exactly the inputs you most want to log.
2. Truncation. If the model hits max_tokens (Anthropic) or max_output_tokens (OpenAI, status: "incomplete"), the JSON stops mid-string. The grammar cannot close braces it never got to emit.
3. Valid but wrong. Gemini's docs say it plainly: output is syntactically correct JSON, so validate the values in your application. A schema-compliant invoice with a hallucinated total is still a hallucinated total. Structured output also does nothing about prompt injection; an attacker who controls the input can steer the values inside a perfectly valid object.
4. Stripped constraints. As shown above, a constraint your SDK silently moved into a description is a suggestion, not a rule. Run your Pydantic validators on every response.
5. Schema rejected or slow. Too many optional fields or unions on Anthropic, more than 1,000 enum values on OpenAI, deep nesting on Gemini: all 400 errors at request time. On Anthropic, the first request with a new schema also pays compile latency, which shows up as a p99 spike after every deploy that changes a model class.
6. Incompatible features. Anthropic's JSON outputs cannot be combined with citations or message prefill; both return errors or are documented as incompatible.

None of this is exotic. It is the discipline platform teams already apply to any third-party API: version the contract (your schema), validate at the edge, alert on the error rate, roll back when a change makes it worse. Logging stop_reason and validation failures as metrics, alongside your LLM traces, turns "the model is flaky" into a number you can watch.

FAQ

Is JSON mode the same as structured outputs?

No. JSON mode only guarantees that the output parses as JSON. Structured outputs constrain decoding to your specific JSON Schema, so required keys, types and enums are enforced. On OpenAI, JSON mode also requires the word JSON in your prompt.

Do structured outputs guarantee valid JSON every time?

They guarantee schema-conformant output when the provider accepts your schema and the response completes normally. Refusals and token-limit truncation can still produce non-conforming output, so always check the stop reason and validate before using the data.

Should I use tool calling or structured outputs for extraction?

Use the response format (structured outputs) when you just want data back, and strict tool calling when the model is choosing among actions. Both use the same constrained decoding on OpenAI and Anthropic, so reliability is similar; the difference is which part of the response carries the JSON.

Which constrained decoding library should I use with a local model?

If you serve with vLLM, start with the default backend setting (auto, which picks a backend per request), and pin xgrammar or guidance explicitly once you know which handles your schemas better. On SGLang, XGrammar is the default. llama.cpp has its own GBNF grammars and JSON Schema conversion, plus an optional llguidance build flag. Outlines is a good fit when you drive Hugging Face Transformers directly from Python.

This quarter I would do three things on any LLM feature that returns data: move every hand-rolled JSON prompt to the provider's structured output mode with schemas generated from Pydantic, add stop_reason and validation-failure counters to the dashboards, and write one eval that runs the schema against twenty awkward real inputs before each schema change ships. The grammar handles syntax now. The rest is still engineering.

Related: How to Build an AI Agent With Tool Calling in Python (2026), OWASP Top 10 for LLMs 2026 and Agentic Apps: 20 Risks, Real Incidents and LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier.