RAG vs Long Context vs Fine-Tuning in 2026: 1M Tokens Isn't Free

Every few months someone tells me RAG is dead because the new model takes a million tokens, and every few months I go and price it. The window really is a million tokens now on most frontier APIs, and you really can paste a whole handbook into the prompt. What you cannot do is assume the model reads all of it equally well, or that you can afford to send it ten thousand times a day. This post compares RAG vs long context vs fine-tuning in 2026 with the numbers that matter: current context windows and prices, what the long-context research actually measured, where retrieval breaks, the per-query cost arithmetic, and the decision framework I use when someone asks which one to build.

Context window sizes and pricing in October 2026

The headline has converged. Anthropic's pricing page says Claude 4.6 and later models (except Haiku 5.5) get the full 1M-token window at standard pricing, so a 900k-token request costs the same per token as a 9k one. OpenAI's GPT-6.1 Sol has a 1,050,000-token window with a 922,000-token input cap, and GPT-5.5 has the same 1.05M window. Google's Gemini 3.1 Pro Preview accepts 1,048,576 input tokens.

The prices are where they diverge. OpenAI and Google both charge more once the prompt is long; Anthropic does not, except on Haiku.

Model	Context window	Input $/MTok	Long-prompt input	Cached input $/MTok	Output $/MTok	Source
Claude Opus 5.5	1M	$4	same rate	$0.20	$20	Anthropic pricing
Claude Sonnet 5.5	1M	$2	same rate	$0.10	$10	Anthropic pricing
Claude Haiku 5.5	tiered at 100K	$0.10	$0.50 over 100K	$0.01	$0.50	Anthropic pricing
GPT-6.1 Sol	1.05M	$2	$4 over 272K	$0.10	$10	OpenAI model page
GPT-5.5	1.05M	$5	$10 over 272K	$0.50	$30	OpenAI pricing
Gemini 3.1 Pro Preview	1,048,576	$2	$4 over 200K	$0.20 ($0.40 over 200K)	$12	Gemini pricing

Two details in the fine print change the arithmetic. On GPT-6.1 Sol, a prompt over 272K tokens pays double input and 1.5 times output for the whole request, not just the tokens past the line. Gemini's context cache adds a storage charge of $4.50 per million tokens per hour on 3.1 Pro, so a cache you keep warm all day is a line item of its own. Embeddings, by contrast, are close to free: OpenAI's text-embedding-3-small is $0.02 per million tokens.

Does a 1M-token context window replace RAG?

For a small, stable corpus, sometimes yes. Anthropic's own retrieval write-up says that if your knowledge base is under 200,000 tokens (about 500 pages), you can skip RAG and put the whole thing in the prompt, especially with prompt caching. That advice was written when 200K was the window; it still holds as a cost-and-quality sweet spot, not a hard limit.

Past that, "replace" runs into two separate problems. The first is accuracy: a model that accepts a million tokens does not attend to them uniformly, which the next section covers. The second is that most production corpora are not 500 pages. A company wiki, a ticket history or a codebase with its dependencies is tens or hundreds of millions of tokens, and no current window takes that. You need a selection step, and a selection step is retrieval whatever you call it. The useful question is not RAG or long context but how much to retrieve and how big a window to retrieve into.

Long-context degradation: needle tests, lost in the middle and context rot

The classic needle-in-a-haystack test hides one fact in filler and asks for it back. Models now ace it, which is why researchers stopped trusting it. A thread of papers since 2023 shows what happens when the task is slightly harder.

Study	Date	What it tested	Finding	Source
Lost in the Middle	2023 (TACL)	multi-doc QA, key-value retrieval	accuracy best when the answer is at the start or end, drops in the middle	arXiv 2307.03172
RULER	2024	17 models, tasks beyond retrieval	near-perfect on vanilla needle tests, but only about half hold up at 32K	arXiv 2404.06654
NoLiMa	Feb 2025, rev. Jul 2025	13 models claiming 128K+, needles with no word overlap	at 32K, 11 models fall below 50% of their short baseline; GPT-4o 99.3% to 69.7%	arXiv 2502.05167
Context Rot (Chroma)	14 Jul 2025	18 models, distractors, LongMemEval	every model did better on ~300-token focused inputs than ~113K full ones	Chroma research
MRCR v2, 8-needle, 1M	5 Feb 2026	Anthropic launch eval	Opus 4.6 76%, Sonnet 4.5 18.5%	Anthropic

NoLiMa is the one I send people. It removes the literal word overlap between question and needle, so the model has to make an association rather than string-match, and that alone collapses most models well before their advertised limit. Chroma's Context Rot report adds the uncomfortable details: a single topically similar distractor hurts, four hurt more, and, counterintuitively, shuffled haystacks scored better than coherent essays across all 18 models. Its LongMemEval comparison is the closest thing to a RAG-versus-stuffing experiment: the same questions, answered from a focused ~300-token context or the full ~113K conversation, and the focused version won for every model.

The 2026 frontier is better. Anthropic reported Opus 4.6 at 76% on the 8-needle 1M MRCR v2, against 18.5% for Sonnet 4.5, which is a real jump in a year. But 76% on a retrieval benchmark is still one miss in four, and it is a vendor-run number on a vendor-chosen test. I treat long-context accuracy as something to measure on my own documents, not something the window size promises.

Why RAG fails: the failure modes nobody demos

Retrieval has its own way of going wrong, and the demos rarely show it. In rough order of how often I see each:

• The right chunk is never retrieved. Pure vector search misses exact identifiers: error codes, SKUs, function names, ticket numbers. Embeddings encode meaning, and ERR_4012 does not have much. This is why hybrid search (keyword plus vector) is the default now, and why the BM25 baseline keeps embarrassing memory products.
• Chunks lose their context. A paragraph that says "revenue grew 3% over the previous quarter" is useless without the company and the quarter. Anthropic's contextual retrieval prepends a short generated summary to each chunk and reported top-20 retrieval failures falling from 5.7% to 3.7% with contextual embeddings, 2.9% adding contextual BM25, and 1.9% with a reranker on top, at about $1.02 per million document tokens to generate the context.
• Approximate indexes drop results under filters. HNSW and IVF filter after the index scan. pgvector's docs give the example plainly: with the default ef_search of 40 and a filter matching 10% of rows, about 4 rows come back on average, not the 10 you asked for. The fix exists (iterative scans since 0.8.0) but you have to know to switch it on.
• Stale or duplicated content. Re-ingesting a changed document without deleting the old chunks leaves two versions competing. This is a data pipeline bug, not a model bug, and it is the most boring and most common.
• Questions that need the whole corpus. "Summarise every incident in Q3" or "which contracts lack a termination clause" are aggregation questions. Top-k retrieval cannot answer them; you need a structured query, a map-reduce pass, or a long window.

Most of these are ordinary engineering: indexing, data freshness, filters, evaluation. That is the quiet point of the whole debate. The teams that do RAG well are the ones that already run databases and pipelines with discipline.

The cost math: long context vs RAG per query

Here is the comparison I run before any design review. Assume a 500K-token corpus that the model must answer questions about, and either stuff it all in or retrieve 20 chunks of 800 tokens (16K tokens, the shape Anthropic found worked better than 10 or 5). Input tokens only, since output is the same either way.

# cost_per_query.py: input-token cost of stuffing vs retrieving, prices as of 2026-10-20
# Sources: platform.claude.com/docs/en/about-claude/pricing,
#          developers.openai.com/api/docs/models/gpt-6.1-sol, ai.google.dev/gemini-api/docs/pricing
CORPUS_TOKENS = 500_000
RAG_TOKENS = 20 * 800            # top-20 chunks of ~800 tokens
QUERIES_PER_DAY = 10_000

# (input $/MTok, cached input $/MTok, long-prompt threshold, long-prompt input, long cached)
PRICES = {
    "claude-sonnet-5.5":      (2.00, 0.10, None,    2.00, 0.10),  # flat across 1M
    "gpt-6.1-sol":            (2.00, 0.10, 272_000, 4.00, 0.20),  # 2x input over 272K
    "gemini-3.1-pro-preview": (2.00, 0.20, 200_000, 4.00, 0.40),  # plus cache storage/hour
}

def input_cost(model, tokens, cached=False):
    base, cache, threshold, long_base, long_cache = PRICES[model]
    if threshold and tokens > threshold:
        base, cache = long_base, long_cache
    return tokens / 1e6 * (cache if cached else base)

for m in PRICES:
    stuff = input_cost(m, CORPUS_TOKENS)
    stuff_cached = input_cost(m, CORPUS_TOKENS, cached=True)
    rag = input_cost(m, RAG_TOKENS)
    print(f"{m:24s} stuff ${stuff:.2f}  cached ${stuff_cached:.3f}  rag ${rag:.3f}"
          f"  | per day: ${stuff*QUERIES_PER_DAY:,.0f} / ${stuff_cached*QUERIES_PER_DAY:,.0f}"
          f" / ${rag*QUERIES_PER_DAY:,.0f}")

Run it and the shape is obvious. On Sonnet 5.5, stuffing costs $1.00 a query uncached, $0.05 on a cache hit, and retrieval $0.032. On GPT-6.1 Sol and Gemini 3.1 Pro the long-prompt surcharge pushes the uncached stuffed query to $2.00. At 10,000 queries a day that is $10,000 a day against $320 for retrieval on Sonnet, before you add Gemini's cache storage. Embedding the whole corpus once costs about one cent at $0.02 per million tokens.

Prompt caching is the honest counter-argument. A cached 500K prefix on Sonnet 5.5 is only about 1.6 times the RAG cost per query, and you skip the entire retrieval pipeline. The catch is that caches expire (Anthropic sells 5-minute and 1-hour writes), writes cost more than base input, and any change to the corpus invalidates the prefix. For a static document queried in bursts, caching is genuinely competitive. For a corpus that changes hourly, it is not. If you are tracking this spend properly, the FOCUS token columns are where it should land.

When is fine-tuning worth it instead of RAG?

Fine-tuning changes behaviour, not knowledge. A 2023 comparison of knowledge injection methods found RAG consistently beat unsupervised fine-tuning for both previously seen and entirely new facts, and that models struggle to learn new facts through fine-tuning at all. That matches what most teams discover the expensive way: you fine-tune on your docs, the model adopts your tone, and it still invents your API parameters.

Where fine-tuning earns its cost is format, style and narrow skills: a fixed output schema, a classification task with thousands of labelled examples, a small model taught to do one job that a large model does expensively. The vendor picture has also shifted. OpenAI's pricing page now says it is winding down its fine-tuning platform: no new users, existing users can train for the coming months, and tuned models stay available until their base models are deprecated. In 2026, fine-tuning increasingly means LoRA on open weights you host yourself, which is a GPU and MLOps commitment, covered in when fine-tuning is worth it.

A decision framework: RAG, long context or fine-tuning

The rule I apply, in order:

1. Corpus under ~200K tokens and fairly static? Put it in the prompt and cache it. Measure accuracy on 50 real questions before building anything else.
2. Corpus larger, or changes daily, or needs per-user permissions? RAG. Start with hybrid search in the database you already run; a Postgres and pgvector pipeline handles millions of chunks without a new vendor, and the case for a dedicated store is in do you need a vector database.
3. Questions need reasoning across many retrieved pieces? Retrieve generously (20 to 50 chunks, reranked) into a large window. This is the hybrid most production systems end up at: retrieval for selection, long context for synthesis.
4. The problem is output format, tone or a narrow repeated task? Fine-tune, ideally a small open model, and still feed it facts through retrieval.
5. Aggregation over everything? Neither; write the SQL or run a batch job.

FAQ

Is RAG dead now that context windows are 1M tokens?

No. Million-token windows make RAG optional for small corpora, but most corpora are far larger than 1M tokens, and research such as NoLiMa and Chroma's Context Rot shows accuracy falling well before the advertised limit. Retrieval also cuts input cost per query by one to two orders of magnitude.

What is context rot?

Context rot is Chroma's name for the measured drop in LLM reliability as input length grows, even on simple tasks. Its July 2025 report tested 18 models and found distractors, low question-to-answer similarity and long inputs all degrade accuracy, and that every model answered better from a focused ~300-token context than from the full ~113K-token one.

Should I fine-tune or use RAG for company documents?

Use RAG for facts. Studies comparing the two find retrieval more accurate for both known and new facts, and RAG lets you update or delete a document without retraining. Fine-tune only to change output style, format or a narrow task, and combine it with retrieval.

Is prompt caching cheaper than RAG?

Per query it can come close. A cached 500K-token prefix on Claude Sonnet 5.5 costs about $0.05 in input against roughly $0.032 for a 16K-token RAG prompt. Caching loses when the corpus changes often, when traffic is too sparse to keep the cache warm, or when the provider adds storage fees.

This quarter I would do three things on any RAG-versus-long-context argument: run the cost script above with my real corpus size and query volume, build a 50-question eval from real user questions and score both approaches on it, and switch any vector-only retrieval to hybrid search with iterative scans enabled. Then decide with numbers instead of with the launch post.

Related: Agent Memory in 2026: Vendors Claim 92.5 on LoCoMo, a Plain Filesystem Scored 74, and Every Memory Product Lost to BM25, LLM Inference in 2026: Prefill Is a Different Machine Now, and the KV Cache Has a Disk Tier and FinOps for AI in 2026: 98% Manage Token Spend, 3 in 4 Can't Prove the Value, GPUs Run at 5%, and FOCUS Gets a Token Column in 1.5.