Why RAG Fails in Production (2026): 7 Failure Points and Real Fixes

Every RAG demo I have seen works, and most RAG systems I have been asked to look at in production quietly don't. The demo has ten clean PDFs and a question the author already knew the answer to. Production has forty thousand documents in six formats, three of them out of date, a permission model that lives in another system, and users who ask things nobody wrote down. This post explains why RAG fails in production in 2026, failure point by failure point: chunking, retrieval misses, stale indexes, permission leaks, the long-context and agentic alternatives, and the evaluation gap that lets all of it ship. For each one I give the fix and the evidence behind it, and where the evidence is thin, I say so.

Where does RAG actually fail?

The most useful map is still a 2024 experience report from Deakin University, Seven Failure Points When Engineering a Retrieval Augmented Generation System, drawn from three deployments in research, education and biomedicine. Its two conclusions are the ones engineers keep relearning: a RAG system can only really be validated while it is operating, and its robustness evolves rather than being designed in up front. The failure points it lists are, roughly: the content was never in the corpus, the right document existed but was not ranked highly enough, it was retrieved but cut when the context was consolidated, it was in the context but the model failed to extract it, the format was wrong, the specificity was wrong, and the answer was incomplete.

Notice how few of those are the model's fault. Three of the seven happen before generation starts. That matches what I see when I debug these systems: by the time someone says "the LLM hallucinated", the actual bug is usually upstream, in ingestion or retrieval, and the model did its best with the wrong paragraphs.

Failure	What it looks like	Fix with evidence	Source
Missing content	Confident answer to a question the corpus cannot answer	Track "no relevant context" rate; let the system refuse	Barnett et al., 2024
Bad chunking	Right document, wrong slice; quality drops past ~2.5k tokens of context	Sentence-level chunks, no overlap, contextual prefixes	Bennani and Moslonka, 2026
Retrieval miss	Gold chunk not in top 20	Contextual embeddings + BM25 + rerank: 5.7% to 1.9% failure	Anthropic
Filtered search returns too little	A 10% filter on HNSW with ef_search 40 yields about 4 rows	Iterative index scans (pgvector 0.8.0+) or partial indexes	pgvector README
Permission leak	One tenant's chunks answer another tenant's question	Permission-aware store, filter in the query, not the prompt	OWASP LLM08:2025
Lost in the middle	Answer present in context but ignored	Fewer, better-ranked chunks; put the best first	Liu et al.
No evaluation	Regressions ship silently	Per-component metrics in CI against a reviewed test set	Ragas paper

Chunking errors: the 2.5k-token context cliff

Chunking is where most teams pick a number from a tutorial (512 tokens, 50 overlap) and never revisit it. A January 2026 systematic analysis of chunking strategies by Bennani and Moslonka is the most direct recent evidence I have found, and it contradicts two pieces of folklore. Overlap gave no measurable benefit and raised indexing cost. Plain sentence chunking was the most cost-effective method and matched semantic chunking up to about 5k tokens. And they found a "context cliff": answer quality dropped once the assembled context went past roughly 2.5k tokens. Treat it as one paper on its own datasets, not a law, but it is a good reason to test your defaults instead of inheriting them.

The other common chunking failure is the orphaned chunk: "Revenue grew 3% over the previous quarter" is useless without knowing which company and which quarter. Anthropic's Contextual Retrieval write-up fixes this by having a model write 50 to 100 tokens of document-level context in front of each chunk before embedding and BM25 indexing. They price the one-off cost at about $1.02 per million document tokens with prompt caching, under their stated assumptions of 800-token chunks and 8k-token documents. That is cheap compared with the engineering time spent tuning chunk sizes by hand.

Retrieval misses: hybrid search and reranking

If the right chunk is not retrieved, nothing downstream can save the answer. Anthropic's numbers are the clearest public measurement of how much the standard fixes buy. Measured as the share of queries where a relevant chunk was missing from the top 20, their baseline failed 5.7% of the time; contextual embeddings brought that to 3.7%; adding contextual BM25 brought it to 2.9%; adding a reranker brought it to 1.9%, a 67% reduction overall.

Two caveats. First, these are Anthropic's datasets and Anthropic's embedding choices; the ordering of the gains (hybrid beats dense-only, reranking helps) has been replicated widely, the exact percentages have not. Second, a reranker is another model call on the hot path, so it costs latency. I would still add BM25 before anything else: it is free, deterministic, and it catches exact identifiers (error codes, SKUs, ticket numbers) that dense embeddings smear. If you are on Postgres, the hybrid search guide shows how to do it without a second database.

Stale indexes and filtered vector search

The quietest failure is the index that no longer matches the source. A policy page is updated, the ingestion job only handles inserts, and the old chunk keeps answering questions for months. Or a document is deleted from the CMS but its embeddings never are, which is both a correctness bug and, if the deletion was a legal request, a compliance one. The fix is boring platform work: an ingestion pipeline keyed on a stable document ID and a content hash, upserts and tombstones rather than appends, a re-embed job when you change embedding model (vectors from two models are not comparable), and a freshness metric per source that someone actually alerts on.

Filtering is the second trap, and it bites exactly when you add tenancy or permissions. The pgvector README spells it out: with approximate indexes, the filter is applied after the index scan, so if a condition matches 10% of rows and HNSW's default hnsw.ef_search is 40, you get about 4 rows back on average. Your "top 10" silently becomes a top 4, and recall drops for the users with the most restrictive permissions. Since 0.8.0, iterative index scans keep scanning until enough rows match (current release is 0.8.7, from 1 October 2026):

-- pgvector 0.8.7, Postgres 17. Tenant-filtered retrieval that does not starve.
SET hnsw.iterative_scan = relaxed_order;   -- keep scanning until LIMIT is met
SET hnsw.max_scan_tuples = 20000;          -- default; raise if filtered recall is still low
SET hnsw.ef_search = 100;                  -- default 40 caps candidates per scan

-- Relaxed order can return rows slightly out of distance order; the
-- materialized CTE plus "+ 0" restores strict order (needed on Postgres 17+).
WITH candidates AS MATERIALIZED (
  SELECT c.id, c.doc_id, c.body, c.embedding <=> $1 AS distance
  FROM chunks c
  WHERE c.tenant_id = $2                 -- permission filter lives in SQL, not the prompt
    AND c.acl_groups && $3::text[]       -- user's groups, resolved at query time
    AND c.deleted_at IS NULL             -- tombstoned chunks never come back
  ORDER BY distance
  LIMIT 20
)
SELECT * FROM candidates ORDER BY distance + 0;

Permission leaks: your retriever is an access-control system

A RAG index is a copy of your data with the access controls stripped off unless you put them back. OWASP made this its own category in the 2025 LLM Top 10, LLM08:2025 Vector and Embedding Weaknesses, which lists unauthorized access, cross-context leaks between tenants sharing one vector database, embedding inversion and data poisoning, and recommends permission-aware vector stores with data partitioned by access level. Its example scenario is the one I worry about most: one group's embeddings returned for another group's queries in a shared database.

The failure also runs the other way: retrieved content is untrusted input. EchoLeak, CVE-2025-32711, rated CVSS 9.3 and patched server-side by Microsoft in 2025, was a zero-click chain in Microsoft 365 Copilot where an attacker's email, pulled in by retrieval, steered the assistant into leaking data from its context. The lesson for anyone building RAG over email, tickets or the web is that whatever can be retrieved can issue instructions, and your retrieval scope is your blast radius. I wrote up the defences in the prompt injection guide.

Practically: resolve the caller's identity and groups at query time, enforce them in the database query (row-level security if you are on Postgres), never rely on a prompt saying "only use documents the user can see", and log which chunk IDs were returned to which user so you can answer the auditor's question later.

Is long context replacing RAG in 2026?

For small corpora, partly yes. Anthropic's own advice is that if your knowledge base is under about 200,000 tokens, you can put the whole thing in the prompt and use prompt caching. Beyond that, the evidence says a long window is not a free substitute for retrieval:

• Lost in the Middle (Liu et al., TACL 2024) showed accuracy is highest when the relevant passage is at the start or end of the context and degrades in the middle.
• Databricks Mosaic Research tested 20 LLMs on long-context RAG from 2,000 to 128,000 tokens; most improved and then declined as context grew, and only a handful of the newest stayed consistent above 64k.
• Chroma's Context Rot report (July 2025) held task difficulty constant across 18 models and still found performance grew less reliable as input length grew.
• Google DeepMind's Self-Route paper (EMNLP 2024) found long context beat RAG on quality when resourced well, but RAG and long context gave identical predictions for over 60% of queries, so routing between them cut cost by 65% on Gemini 1.5 Pro and 39% on GPT-4o at comparable quality.

My reading: long context moves the cliff, it does not remove it, and you pay per token every time. I go deeper on the trade in RAG vs long context vs fine-tuning.

Agentic retrieval, where a model plans queries, searches iteratively and decides when it has enough, is the other 2026 answer to retrieval misses. It works, and it is expensive. Anthropic's multi-agent research system post reports a lead-plus-subagents setup beating a single agent by 90.2% on their internal research eval, while noting that agents use about 4 times the tokens of chat and multi-agent systems about 15 times, and that token usage alone explained 80% of the performance variance on the BrowseComp evaluation. In other words, a lot of the gain is buying more retrieval. Use it for questions that are worth the bill, and keep single-shot retrieval for the FAQ traffic.

Evaluation gaps: why bad RAG ships

Most of the failures above are invisible in a demo and obvious in a metric, which is why the real root cause is usually the absence of an evaluation loop. You need to measure retrieval and generation separately: did the gold chunk come back (context recall), how much junk came with it (context precision), did the answer stay inside the retrieved context (faithfulness), and did it address the question (answer relevancy).

LLM judges make this affordable but not free of error. The original Ragas paper measured agreement with human annotators on its WikiEval set: 0.95 for faithfulness, 0.78 for answer relevance and 0.70 for context relevance, against 0.72, 0.52 and 0.63 for simply asking ChatGPT for a 0 to 10 score. That is good for faithfulness and mediocre for relevance. Zheng et al. showed LLM judges reach over 80% agreement with humans but carry position, verbosity and self-enhancement biases. So calibrate the judge against a few dozen human labels, pin the judge model, and treat a score change after a judge upgrade as a measurement change, not a product change. The step-by-step version, with a CI gate, is in my Ragas evaluation guide, and the general argument is in evals are the new unit tests.

FAQ

Why does my RAG system hallucinate even with retrieved context?

Usually because the retrieved context was wrong or incomplete, not because the model ignored good context. Check context recall first: if the gold chunk is missing, the model is filling gaps. If recall is high and faithfulness is low, then look at prompt design, context ordering and context length.

What chunk size should I use for RAG in 2026?

Start with sentence-based chunks of a few hundred tokens without overlap, add a short contextual prefix per chunk, and keep the assembled context modest; one 2026 study saw quality fall past roughly 2.5k tokens. Then tune against your own test set, because the right answer depends on your documents and questions.

Is RAG dead now that models have 1M-token context windows?

No. For corpora under about 200,000 tokens, stuffing the context with prompt caching is often simpler. Above that, long context is slower, costs more per query, and degrades with length, and it does nothing for permissions or freshness. Many teams route between the two.

How do I stop RAG from leaking documents across users?

Enforce access control inside the retrieval query using the caller's identity, partition by tenant, and never trust the prompt to filter. Watch for approximate-index post-filtering returning too few rows, and log which chunks each user received.

If I were inheriting a RAG system this quarter, I would not touch the model first. I would build a 100-question reviewed test set, measure context recall per source, add BM25 and a reranker if recall is below what the business can live with, put tenant and ACL filters into the SQL with iterative scans enabled, and wire a freshness alert on every ingestion source. Then I would gate merges on the eval scores. None of that is new; it is the same pipeline, data and access discipline platform teams have always done, applied to a system that fails politely instead of loudly.

Related: How to Build a RAG Pipeline With Postgres and pgvector (2026), Vector Databases in 2026: Do You Need One, or Is pgvector Enough? and How to Tune pgvector Performance: HNSW, ef_search and Quantization (2026).