Why your RAG is retrieving the wrong documents

A RAG system has two failure modes: retrieval and generation. Teams spend almost all their time on generation — prompt engineering, model selection, fine-tuning — because it’s the visible part. The document that comes back wrong is obvious. What isn’t obvious is that the LLM never had a chance, because the retrieval sent it the wrong context in the first place.

Industry analysis in 2026 consistently shows that when RAG fails, the failure point is retrieval 73% of the time, not generation. The LLM is being asked to answer a question using evidence it was never given. No amount of reasoning fixes that.

The embedding-gap problem

Vector search works by embedding both the query and the documents in the same high-dimensional space. The system retrieves the documents whose embeddings are closest to the query embedding. The problem: semantic similarity is not the same as relevance.

A query like “How do we stop customers from being kicked out?” embeds differently than the document that answers it — “Session timeout behavior and re-authentication flows.” The embedding model trained on generic text may place them far apart in the vector space, even though a human reading both would see they’re about the same thing.

This gets worse as your document corpus grows. In a corpus of 1,000 documents, nearest-neighbor search finds relevant documents with decent probability. In a corpus of 100,000, the probability of retrieving the right document in the top-5 results drops significantly — the vector space becomes sparse, and the query embedding is more likely to be closer to the wrong cluster of documents.

Chunking destroys context

The standard advice is to chunk documents into 512-token segments, embed each chunk, and retrieve the top-k. This is wrong for most enterprise use cases.

Consider a document structured like this:

## Refund Policy
Our refund policy applies to all purchases made after January 1, 2024.
Customers may request a full refund within 30 days of purchase.
For enterprise customers, the refund window is 90 days.
...

A chunk broken at the wrong point might extract only “Customers may request a full refund within 30 days of purchase.” without the enterprise exception. Retrieval returns this to the LLM, which then confidently gives the wrong answer to an enterprise customer asking about their refund window.

The root cause: chunking is a preprocessing decision made without knowledge of the queries that will be run against it. You chunk for your embedding model’s context window, not for the questions your users actually ask.

What “retrieval is working” actually looks like

There’s no single metric that tells you if your retrieval is working. The proxies teams use — recall@k, hit rate, MRR — measure how often the right document appears in the top-k results. But they don’t measure whether the retrieved documents are sufficient to answer the question.

The actual test is whether your RAG system produces correct answers on a representative set of questions, with retrieval isolated from generation. Run the retrieved documents to a human evaluator without the LLM’s answer. If the human can’t answer correctly from those documents alone, the retrieval failed — even if the final LLM answer happens to be right.

We built this diagnostic into every RAG engagement from the start, before tuning anything else. It’s the first eval we write.

The five retrieval failure patterns we see most

1. Semantic mismatch. The query and the relevant document use different vocabulary. The embedding space doesn’t bridge them.

2. Chunk boundary artifacts. Relevant context is split across chunks, and neither chunk is retrieved alone.

3. Top-k too small. The retrieved documents don’t contain the answer, but document 11 does. Increasing k past 10-20 often reveals this.

4. Context starvation. The retrieved documents are individually relevant but collectively insufficient — answering requires synthesizing across documents that aren’t retrieved together.

5. Metadata filtering mismatch. Hybrid retrieval (vector + keyword) applies metadata filters that exclude the relevant document for reasons unrelated to the query semantics.

How to fix it

For semantic mismatch: hybrid retrieval + query expansion. Pure vector search fails when vocabulary diverges. Combine vector search with BM25 keyword matching — the intersection is almost always more relevant than either alone. Add query expansion: rewrite the user’s query into 2-3 reformulations, embed all of them, and take the union of retrieved documents before re-ranking.

For chunk boundary artifacts: parent-document retrieval. Instead of chunking at a fixed token boundary, chunk at semantic boundaries — paragraph or section breaks. Then retrieve at the chunk level, but pass the parent document (or a wider window) to the LLM. This preserves context without flooding the context window.

For top-k too small: set k high, then re-rank. Retrieve 20-50 candidates, then use a cross-encoder re-ranker (a small model that scores each query-document pair for relevance) to select the final 3-5. This costs more latency but dramatically improves retrieval quality.

For context starvation: hierarchical retrieval. Query a summary index first (chunk-level summaries of each document), retrieve the most relevant documents, then retrieve chunks within those documents. This works when answers require synthesizing across a document rather than a single chunk.

For metadata filtering mismatch: rethink the filter. If you’re filtering by date, document type, or department, check whether the metadata actually correlates with query intent. In most enterprise corpora, it doesn’t — the filter removes relevant documents that happen to have the wrong metadata tag.

The retrieval harness

The same principle from our harness engineering post applies here: build the measurement before you tune. For RAG, that means writing an eval suite of real queries with human-verified expected documents before you optimize retrieval. Track recall@k per query category, not just aggregate. Find the failure patterns specific to your corpus, then pick the fix that addresses the dominant failure mode.

The LLM is the easy part. Retrieval is where production RAG actually fails.

What to do next

• Write 20 real queries from your users. For each, identify which document contains the answer.
• Run those queries against your current retrieval and compute recall@10. If it’s below 80%, your retrieval needs work.
• For each failure, classify it: semantic mismatch, chunk boundary, top-k, context starvation, or metadata. Fix the dominant class first.
• If your corpus is larger than 10,000 documents, consider hierarchical retrieval or a summary index before optimizing chunking.

The eval-driven approach works for RAG the same way it works for everything else in AI engineering: you can’t fix what you can’t measure.

---

See how Ankor applies this retrieval-first thinking in production: the Compass career recommendations case study uses hybrid retrieval to surface the right career matches from a complex assessment corpus. Or read more about our RAG implementation service.