vvek

Better retrieval for your AI app: give every chunk its context, then combine two kinds of search

Why chunks lose meaning in RAG, how contextual retrieval, BM25 and reranking cut missed results in Anthropic's tests, and a checklist to try on your own data.

Retrieval-augmented generation (RAG) has a quiet weak spot. Before anything is searched, your documents are cut into chunks, and a chunk is often a poor witness to what it came from.

Take a sentence like "Revenue grew 3% over the previous quarter." Inside its report, it is perfectly clear. On its own, in a vector index, it does not say which company, or which quarter. A question about one company's revenue growth may never reach it, or may match the wrong company's chunk equally well.

Anthropic described a fix for this in Introducing Contextual Retrieval. The idea is small, and you can try most of it this week.

Step 1: give each chunk a sentence of context

Before you embed a chunk, prepend a short note that places it in its document: which document, which section, what it is about. The article uses a model to read the full document and the chunk, then writes a note that's usually 50 to 100 tokens.

A chunk like the revenue sentence would start with something such as "From a quarterly report for ACME Corp, second quarter, results section." Then the text follows. That combined text is what you embed, and what you index for keyword search.

The note is written once, when you index, not on every query.

Tip

The article suggests that a custom prompt for the context note, for example one that includes a glossary of your domain's terms, may beat the generic one. Start generic, then look at your worst misses and adjust.

Step 2: add keyword search next to embeddings

Embeddings are good at meaning and weaker at exact strings. A question that contains an error code, a product identifier or a function name wants a literal match. The article's example is an identifier like "TS-999".

BM25 is a classic ranking function for keyword search, and it handles exactly that case. The article runs both searches, then merges the two result lists with rank fusion and removes duplicates. The context note also goes into the BM25 index, so the keyword search benefits too.

If your data lives in Postgres, you may already have what you need for the keyword side through full-text search. Our post on vector search with pgvector covers the vector half.

Step 3: fetch wide, rerank, pass few

Merging gives you a long candidate list. A reranker is a model that reads the question together with each candidate and scores how relevant it is. In the article, the first pass retrieves the top 150 chunks, the reranker scores them, and the top 20 go to the model that writes the answer.

The same article compared passing 5, 10 and 20 chunks, and 20 did best. It also warns that more text can distract the model, so there is a ceiling. Treat 20 as a place to start measuring, not as a rule.

flowchart LR
  A[Document] --> B[Split into chunks]
  B --> C[Add context note]
  C --> D[Embedding index]
  C --> E[BM25 index]
  Q[Question] --> D
  Q --> E
  D --> F[Merge with rank fusion]
  E --> F
  F --> G[Rerank top 150]
  G --> H[Top 20 to the model]

What the numbers say

The article measures how often the right chunk is missing from the top 20 results (one minus recall@20), averaged across the domains it tested, with Gemini Text 004 embeddings:

Setup Missed in top 20 Reduction
Plain embeddings 5.7% baseline
Contextual embeddings 3.7% 35%
Contextual embeddings plus contextual BM25 2.9% 49%
Plus reranking 1.9% 67%

Each step helps, and the steps stack. The article also notes that contextualizing improved every embedding model it tried, and singles out Gemini and Voyage embeddings as especially effective.

Note

These are one team's results on their own datasets. The article itself advises running evaluations on your own use case, because the gains may not transfer directly. Read the table as a reason to test, not a promise.

A checklist for this week

  1. Collect 30 to 50 real questions with the document you expect each to find. This is your measuring stick. Our post on evals for AI agents shows how to keep such a set small and useful.
  2. Measure your baseline: for each question, is the right chunk in the top 20? Write down the percentage.
  3. Look at the misses. If many are chunks that lack a name, date or section, context notes will likely help. If many are exact identifiers, add keyword search first.
  4. Add the context note at indexing time and re-embed. Re-measure.
  5. Add BM25 and merge with rank fusion. Re-measure.
  6. Add a reranker only if the misses remain, since it adds a model call to every query.
  7. Vary the chunk count (for example 5, 10, 20) and see where answer quality stops improving.

Change one thing at a time. If a step doesn't help your data, skip it and keep things simple.

Things to watch

Chunk size, boundaries and overlap also affect retrieval, according to the article, so the context note does not replace good chunking. If you can, mark the added context so the answering model can tell it apart from the original text; the article notes that may improve responses.

Finally, remember that a longer prompt is not the goal. Better retrieval means the model sees fewer, more relevant passages, which fits the ideas in context engineering for agents.

Sources

Keep reading

A short email when we publish. Confirm once, unsubscribe any time.