Your RAG System Has a Retrieval Problem

Your RAG System Has a Retrieval Problem

A retrieval-augmented system gives a wrong answer, and the instinct is to blame generation. Change the prompt. Try a bigger model. Turn the temperature down. Most of the time the model never saw the right document, and none of that helps.

Separate the two failures first

Before touching anything, measure retrieval and generation independently. For each failing question, check whether the correct chunk was in the retrieved set at all. That single check partitions your failures into two piles with completely different fixes, and it takes an afternoon.

Usually the retrieval pile is the larger one, by a wide margin.

Chunking destroys things quietly

A chunk boundary through the middle of a table, or between a claim and its qualifier, produces a chunk that is retrievable and wrong. Fixed-size chunking with no respect for document structure is the default in most tutorials and is the source of a startling share of production errors.

Dense retrieval alone misses exact terms

Embeddings are good at meaning and bad at identifiers. Part numbers, error codes, proper nouns and version strings are exactly what users search for and exactly what dense retrieval blurs. Hybrid search (dense plus lexical) is not a refinement, it is usually the fix.

Reranking earns its latency

Retrieve broadly, then rerank precisely. A cross-encoder over the top fifty candidates costs tens of milliseconds and routinely does more for answer quality than a model upgrade costing ten times as much per token.

The order to work in

  • Measure retrieval recall separately from answer quality
  • Fix chunking against the structure of your actual documents
  • Add lexical search alongside dense
  • Add a reranker
  • Only then look at the generation model

Teams that follow that order tend to stop after step three.

Keep reading

Read the Loss Curve: A Field Guide to Failed Runs
Read the Loss Curve: A Field Guide to Failed Runs

Loss flat, loss spiking, training down and validation up. Each shape has a short list of causes, and guessing is not one of the steps.

Gradient Boosting Is Still the Answer for Tabular Data
Gradient Boosting Is Still the Answer for Tabular Data

Reaching for a neural network on tabular data is a common way to look junior in a system design interview.

What a Tailored Resume Should Actually Mean
What a Tailored Resume Should Actually Mean

Not keyword stuffing, and not a new resume per job. Reordering the truth, with the numbers attached.