How to Cut LLM API Costs: Prompt Caching, Batching, Routing (2026)

This guide is for engineers who own an LLM feature and a bill that grows faster than traffic. By the end you will have per-route cost and cache-hit logging, a cache-friendly prompt layout, working caching code for Claude, OpenAI and Gemini, a batch pipeline for work nobody waits on, and a simple router with output caps. Code was checked against anthropic 2.3.0, openai 3.42.0 and google-genai 2.34.0 on Python 3.12, and prices against the vendors' pricing pages on 31 October 2026. You need an API key for at least one provider and a workload with a repeated prefix of at least a few thousand tokens. For the theory behind the multipliers, read the companion post, prompt caching explained.

Step 1: Measure cost and cache hits per route

Do not change anything until you can see what each route costs. Every provider returns token counts on the response, and that is the only reliable source of truth for whether caching works. Pin the SDKs first:

python3.12 -m venv .venv && source .venv/bin/activate
pip install anthropic==2.3.0 openai==3.42.0 google-genai==2.34.0

Then wrap your calls so every response logs its route, model and usage. For Claude, the full prompt size is input_tokens + cache_creation_input_tokens + cache_read_input_tokens; input_tokens alone is only the uncached tail.

# costlog.py: per-request cost from the usage block (Claude).
# Prices: USD per million tokens from platform.claude.com pricing, 2026-10-31.
PRICES = {
    #                  input  5m write  read  output
    "claude-sonnet-5-5": (2.00, 2.50, 0.10, 10.00),
    "claude-haiku-5-5":  (0.10, 0.125, 0.01, 0.50),   # prompts up to 100K tokens
    "claude-opus-5-5":   (4.00, 5.00, 0.20, 20.00),
}

def request_cost(model: str, usage) -> dict:
    p_in, p_write, p_read, p_out = PRICES[model]
    cost = (usage.input_tokens * p_in
            + usage.cache_creation_input_tokens * p_write   # 5-minute writes
            + usage.cache_read_input_tokens * p_read
            + usage.output_tokens * p_out) / 1_000_000
    prompt = (usage.input_tokens + usage.cache_creation_input_tokens
              + usage.cache_read_input_tokens)
    return {
        "usd": round(cost, 6),
        "hit_rate": usage.cache_read_input_tokens / prompt if prompt else 0.0,
        "prompt_tokens": prompt,
        "output_tokens": usage.output_tokens,
    }

If you use 1-hour writes, usage.cache_creation splits writes into ephemeral_5m_input_tokens and ephemeral_1h_input_tokens; price the 1-hour part at 2x input. Send the dict to whatever you already use for metrics. If you already run a gateway, the LiteLLM proxy guide shows how to get spend per key and team without touching app code. A week of this data tells you which routes are input-heavy (caching wins) and which are output-heavy (routing and caps win).

Step 2: Make the prompt prefix byte-stable

Caching is a prefix match: one changed byte invalidates everything after it. Anthropic renders tools, then system, then messages, so put the content that never changes first and the per-request content last. The usual offenders are timestamps and IDs in the system prompt and nondeterministic tool serialisation.

# prompt_build.py: build a deterministic prefix.
import json
from pathlib import Path

SYSTEM_PROMPT = Path("prompts/support_v14.md").read_text()   # frozen, versioned file

def build_tools(tool_specs: list[dict]) -> list[dict]:
    # Same order every time, same key order inside each schema.
    tools = sorted(tool_specs, key=lambda t: t["name"])
    return json.loads(json.dumps(tools, sort_keys=True))

def build_user_turn(question: str, today: str, request_id: str) -> str:
    # Volatile values go here, after the cached prefix, never in SYSTEM_PROMPT.
    return f"<context date='{today}' request='{request_id}'/>\n{question}"

Two more rules: give every user the same tool set (pass modes as message content instead of swapping tools), and keep one model per conversation, because caches are scoped to a model.

Step 3: Add prompt caching to Claude with cache_control

Anthropic offers automatic caching (one top-level cache_control) and explicit breakpoints on blocks. For a large shared system prompt with a varying question, put an explicit breakpoint on the last system block so the question is never part of the cached key:

# claude_cached.py: anthropic==2.3.0
import anthropic
from prompt_build import SYSTEM_PROMPT, build_user_turn
from costlog import request_cost

client = anthropic.Anthropic()          # reads ANTHROPIC_API_KEY
MODEL = "claude-sonnet-5-5"

def ask(question: str, today: str, request_id: str):
    resp = client.messages.create(
        model=MODEL,
        max_tokens=1024,                 # a ceiling sized for this route, see Step 7
        system=[{
            "type": "text",
            "text": SYSTEM_PROMPT,       # must be >= the model's minimum (512 on Sonnet 5.5)
            "cache_control": {"type": "ephemeral"},   # 5-minute TTL; add "ttl": "1h" for gaps of 5 to 60 min
        }],
        messages=[{"role": "user", "content": build_user_turn(question, today, request_id)}],
    )
    print(request_cost(MODEL, resp.usage))
    return resp

For a multi-turn chat or agent loop, add top-level automatic caching as well, which moves a breakpoint forward as the conversation grows. Keep the explicit marker on the system prompt so the expensive shared part always has a read point:

resp = client.messages.create(
    model=MODEL,
    max_tokens=2048,
    cache_control={"type": "ephemeral"},   # automatic: caches up to the last cacheable block
    system=[{"type": "text", "text": SYSTEM_PROMPT,
             "cache_control": {"type": "ephemeral"}}],
    tools=tools,                           # from build_tools(), identical every turn
    messages=history,                      # append-only; never edit earlier turns
)

Limits from the prompt caching docs: at most four breakpoints per request; each breakpoint looks back at most 20 blocks for an earlier entry; minimum cacheable length is 512 tokens on the newest models, 1,024 on Opus 4.8, Sonnet 5 and Sonnet 4.6, 2,048 on Opus 4.7, and 4,096 on Opus 4.6 and Haiku 4.5. Shorter prefixes are silently processed without caching.

Step 4: Verify the cache actually hits

Send the same prefix twice inside five minutes and compare the usage fields. Make this an integration test so a future prompt change cannot quietly break it.

# test_cache.py: run with pytest; costs two small real requests.
from claude_cached import ask

def test_second_call_reads_cache():
    first = ask("How do I reset my password?", "2026-10-31", "req-1")
    second = ask("How do I change my email?", "2026-10-31", "req-2")
    assert first.usage.cache_creation_input_tokens > 0 or first.usage.cache_read_input_tokens > 0
    assert second.usage.cache_read_input_tokens > 0, "prefix changed between calls"

The shape to expect (exact numbers depend on your prompt): on the first call cache_creation_input_tokens is roughly the size of the system prompt and cache_read_input_tokens is 0; on the second call those swap, and input_tokens is just the user turn. If both are 0, you are under the minimum length. If the first number shows up on every call, something in the prefix is changing. On the Claude API there is also cache diagnostics, now stable: pass a diagnostics object with the previous response's id as previous_message_id and the response names where the two requests diverged. The old cache-diagnosis-2026-04-07 beta header is no longer required.

Step 5: Prompt caching on OpenAI and Gemini

OpenAI, GPT-5.6 and later. Caching is implicit by default, with one breakpoint at the end of the latest message. Writes now cost 1.25x for the default 30-minute lifetime, so for a shared prefix with a varying question, switch to explicit mode and mark the end of the shared part. Field names below are from the prompt caching guide and the 3.42.0 SDK types:

# openai_cached.py: openai==3.42.0, Responses API
from openai import OpenAI
from prompt_build import SYSTEM_PROMPT

client = OpenAI()

resp = client.responses.create(
    model="gpt-6.1-terra",
    prompt_cache_options={"mode": "explicit"},     # no implicit breakpoint; ttl defaults to "30m"
    input=[
        {"role": "developer", "content": [{
            "type": "input_text",
            "text": SYSTEM_PROMPT,                   # breakpoints cannot go in top-level instructions
            "prompt_cache_breakpoint": {"mode": "explicit"},
        }]},
        {"role": "user", "content": [{"type": "input_text", "text": "How do I reset my password?"}]},
    ],
    max_output_tokens=1024,
)
d = resp.usage.input_tokens_details
print("read:", d.cached_tokens, "write:", d.cache_write_tokens, "total in:", resp.usage.input_tokens)

Uncached input is input_tokens - cached_tokens - cache_write_tokens. On models before GPT-5.6 there is no write charge and no breakpoint; instead set a stable prompt_cache_key per shared prefix and, where supported, prompt_cache_retention="24h". The guide notes caches live on individual machines and that more than about 15 requests per minute on one key can overflow, so split very busy keys.

Gemini. Implicit caching is on by default for Gemini 2.5 and newer, with no guaranteed saving; check usage_metadata. For a guaranteed discount on Gemini 3 and 2.5 models, create an explicit cache with the generateContent API (beta, not supported on Gemini 4 Argon):

# gemini_cached.py: google-genai==2.34.0
from google import genai
from google.genai import types
from prompt_build import SYSTEM_PROMPT

client = genai.Client()                            # reads GEMINI_API_KEY
MODEL = "models/gemini-3.8-flash"

cache = client.caches.create(
    model=MODEL,
    config=types.CreateCachedContentConfig(
        display_name="support-v14",
        system_instruction=SYSTEM_PROMPT,
        ttl="3600s",                               # storage is billed for the whole TTL
    ),
)
resp = client.models.generate_content(
    model=MODEL,
    contents="How do I reset my password?",
    config=types.GenerateContentConfig(cached_content=cache.name),
)
print(resp.usage_metadata.cached_content_token_count, resp.usage_metadata.prompt_token_count)
client.caches.delete(name=cache.name)              # stop paying storage when the job ends

Check the storage maths before you commit. On Gemini 3.8 Flash (pricing, through 31 December 2026) input is $0.75 per million, cached input $0.075 and storage $0.50 per million per hour. A 100,000-token cache costs $0.05 an hour to keep and saves $0.0675 per hit, so one hit an hour already covers it.

Step 6: Move offline work to a batch API

Anything nobody is waiting on (nightly classification, evals, backfills, embeddings) belongs on a batch API: Anthropic, OpenAI and Google all charge 50% of the standard price. Anthropic's batch docs allow 100,000 requests or 256 MB per batch, with most finishing within an hour and a 24-hour expiry. Caching multipliers stack with the batch discount, but hits are best effort because requests run concurrently. Anthropic's own advice is to warm the shared prefix with a single 1-hour-TTL request first, then submit the rest:

# claude_batch.py: anthropic==2.3.0
import time, anthropic
from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
from anthropic.types.messages.batch_create_params import Request
from prompt_build import SYSTEM_PROMPT

client = anthropic.Anthropic()
SYSTEM = [{"type": "text", "text": SYSTEM_PROMPT,
           "cache_control": {"type": "ephemeral", "ttl": "1h"}}]   # outlives the batch run

def req(i: int, ticket: str) -> Request:
    return Request(custom_id=f"ticket-{i}", params=MessageCreateParamsNonStreaming(
        model="claude-haiku-5-5", max_tokens=256, system=SYSTEM,
        messages=[{"role": "user", "content": f"Classify this ticket:\n{ticket}"}]))

def wait(batch_id: str):
    while client.messages.batches.retrieve(batch_id).processing_status != "ended":
        time.sleep(60)

tickets = open("tickets.txt").read().split("\n---\n")
warm = client.messages.batches.create(requests=[req(0, tickets[0])])   # writes the 1h cache
wait(warm.id)
main = client.messages.batches.create(requests=[req(i, t) for i, t in enumerate(tickets[1:], 1)])
wait(main.id)

for r in client.messages.batches.results(main.id):    # any order: key by custom_id
    if r.result.type == "succeeded":
        u = r.result.message.usage
        text = next((b.text for b in r.result.message.content if b.type == "text"), "")
        print(r.custom_id, u.cache_read_input_tokens, text)
    else:
        print(r.custom_id, r.result.type)              # errored / canceled / expired: resubmit

OpenAI's Batch API takes a JSONL file (one model per file, up to 50,000 requests and 200 MB) and a 24h completion window:

# requests.jsonl: one request per line
cat > requests.jsonl <<'EOF'
{"custom_id": "ticket-1", "method": "POST", "url": "/v1/responses", "body": {"model": "gpt-6-luna", "input": "Classify: printer on fire", "max_output_tokens": 64}}
{"custom_id": "ticket-2", "method": "POST", "url": "/v1/responses", "body": {"model": "gpt-6-luna", "input": "Classify: invoice missing", "max_output_tokens": 64}}
EOF
FILE_ID=$(curl -s https://api.openai.com/v1/files -H "Authorization: Bearer $OPENAI_API_KEY" \
  -F purpose=batch -F file=@requests.jsonl | jq -r .id)
curl -s https://api.openai.com/v1/batches -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d "{\"input_file_id\": \"$FILE_ID\", \"endpoint\": \"/v1/responses\", \"completion_window\": \"24h\"}" | jq '{id, status}'
# Poll GET /v1/batches/{id} until "completed", then download output_file_id via /v1/files/{id}/content.

Step 7: Route by task and cap output

Caching does nothing for output tokens, and output is priced at five times input on every model in this guide. Two levers remain: send each task to the cheapest model that passes your evals, and stop paying for words nobody reads.

# router.py: per-route model, effort and output ceiling (anthropic==2.3.0).
import logging, anthropic

client = anthropic.Anthropic()
log = logging.getLogger("llm.router")

ROUTES = {
    #  route          model                effort    max_tokens
    "classify":   ("claude-haiku-5-5",   "low",    256),
    "support":    ("claude-sonnet-5-5",  "medium", 1024),
    "code_review":("claude-opus-5-5",    "high",   8000),
}

def call(route: str, system: list, messages: list):
    model, effort, cap = ROUTES[route]
    resp = client.messages.create(
        model=model, max_tokens=cap, system=system, messages=messages,
        output_config={"effort": effort},   # lower effort means fewer thinking tokens
    )
    if resp.stop_reason == "max_tokens":
        log.warning("route %s hit its cap; raise it or tighten the prompt", route)
    return resp

max_tokens is a ceiling, not a target: you pay for tokens generated, so a cap only saves money on runaway answers, and a cap that is too low costs a retry. Pin effort per route rather than per request, because changing it invalidates part of the cache. Do not bounce one conversation between models either: each model keeps its own cache. Route at the task boundary, and validate each route with an eval before switching. The AI gateway comparison covers where to put this logic if several services share it.

Troubleshooting

Symptom	Likely cause	Fix
cache_read_input_tokens and cache_creation_input_tokens both 0	Prefix below the model's minimum (512 to 4,096 tokens on Claude)	Grow the stable part past the threshold, or accept no caching
cache_creation_input_tokens large on every call, reads 0	Timestamp, UUID or user data in the system prompt	Move volatile values into the user turn after the breakpoint
Hit rate fell after a deploy	Tool order or JSON key order changed	Sort tools by name, json.dumps(..., sort_keys=True), add the Step 4 test
Parallel fan-out all miss	An entry is readable only after the first response starts	Send one request, wait for its first token, then fan out
Hits vanish after a few idle minutes	5-minute TTL expired	Use "ttl": "1h" for 5 to 60 minute gaps
Long agent turns stop hitting	20-block lookback window exceeded	Add an intermediate breakpoint within 20 blocks of the last one
Same prompt misses across teams	Caches are isolated per workspace on the Claude API	Run shared traffic from one workspace
OpenAI cached_tokens low at high traffic	Overflow beyond about 15 requests per minute per prefix	Shard prompt_cache_key (pre-5.6) or use explicit breakpoints (5.6+)
Gemini bill rose after explicit caching	Storage billed for the full TTL	Shorten ttl, delete caches when jobs finish
Batch results show few cache reads	Batch hits are best effort	Warm the prefix with a single 1-hour request first

FAQ

Does lowering max_tokens reduce LLM API cost?

Only when the model would otherwise write more than the cap. You are billed for tokens generated, not for the ceiling. A cap protects you from runaway answers; prompts that ask for short answers and lower effort do more for the average bill.

Can I use prompt caching and the Batch API together?

Yes. Anthropic states the multipliers stack, and OpenAI and Google list cached-input prices for batch. Hits inside a batch are best effort, so warm the shared prefix first.

Does switching models break the prompt cache?

Yes. Caches are scoped to a model, so a router that alternates models within a conversation pays full input price on each switch. Route per task or per conversation, not per turn.

How do I see my cache hit rate?

Divide cached tokens by total prompt tokens from each response: cache_read_input_tokens on Claude, input_tokens_details.cached_tokens on OpenAI, cached_content_token_count on Gemini. Aggregate per route and alert on drops.

In production, the first thing I check on any surprising LLM bill is the ratio of cache reads to total prompt tokens for the top three routes, because a broken prefix is the most common and most invisible regression. Then I look for synchronous calls that nobody waits on, and only after that at model choice. The official references are Anthropic's pricing and caching docs, OpenAI's pricing and caching guide, and Google's context caching and Batch API pages.

Related: How to Set Up LiteLLM Proxy, FinOps for AI in 2026 and AI Gateways in 2026.