Summary

Chunks are the unit of retrieval, so chunk boundaries determine what can be found. Fixed-size splitting cuts through sentences, tables, and code, and the resulting wrong answers look like prompt problems. Prefer boundaries the document provides, keep atomic units whole, prepend the heading path to every chunk, and attach position and provenance metadata at index time because it cannot be recovered later.

An embedding places text in a space where similar meanings are close, which is why it finds paraphrases and why it cannot distinguish 220C from 180C. Choose a model on the retrieval subset of a benchmark and then on your own fifty labelled queries. Normalize consistently and use the metric the model was trained for. Cache on content and model together, and remember that changing the model means re-embedding everything, so version the index and reindex blue-green with an evaluation step before switching.

Vector search finds meaning and misses exact identifiers, keyword search does the reverse, and hybrid with reciprocal rank fusion is the standard answer. Pre-filter rather than post-filter, always for permissions and selectivity. Retrieve wide and rerank narrow, because a cheap retriever plus a reranker often beats an expensive retriever alone. Use two different top_k values, one for candidates and one for what reaches the model.

A vector store is a Retriever implementation behind an interface, with filters inside the search and a delete-by-document operation. Index choice is a three-way tradeoff between recall, latency, and memory, and the recall you are losing is measurable against a flat index on a sample.

Assembly matters: ordering interacts with position effects, duplicates waste slots, and citation labels must be validated because a model can invent them. Query transformation helps and costs latency on every request, so measure each one. And retrieval is the wrong tool for aggregation, exhaustive listing, structured filtering, and recency, all of which should be routed elsewhere.

Diagnose before tuning. Log the retrieved, reranked, and final chunk identifiers so a wrong answer can be attributed to retrieval, assembly, or generation. Recall@50 is the ceiling on everything downstream, and the gap between recall@50 and recall@5 tells you whether to fix retrieval or ranking. Build the labelled query set first, change one thing at a time, and slice by query category.

Key takeaways

• Chunk boundaries are a silent cause of wrong answers, and the symptom looks like a prompt problem
• Never split a table, a code block, or any other atomic unit
• Prepend the heading path to every chunk
• Similarity is not truth: 220C and 180C are very close vectors
• The leaderboard average is not the retrieval score, and your own fifty queries beat both
• Changing the embedding model means re-embedding the whole corpus
• Cache embeddings keyed on content and model together
• Vector search misses exact identifiers, which is why hybrid exists
• Fuse by rank, not by score
• Post-filtering a selective filter returns almost nothing
• Retrieve wide, rerank narrow, send few
• Index choice is recall against latency against memory, choose two
• Validate every citation the model returns
• Recall@50 sets the ceiling on everything downstream
• A gap between recall@50 and recall@5 is a reranking problem, not a retrieval one
• Build the labelled query set before touching a single parameter

Common mistakes to remember

• Fixed-size chunking on structured documents
• Splitting a table away from its header row
• Chunking in characters when the budget is in tokens
• Heavy overlap used to compensate for bad boundaries
• Discarding heading and provenance metadata at chunk time
• Choosing an embedding model from a leaderboard average
• Mixing vectors from two models in one index
• Caching embeddings without the model name in the key
• Adding cosine and BM25 scores together
• Post-filtering by permission, which leaks result counts
• Using one top_k for both candidates and final context
• Assuming an approximate index has the recall you hoped for
• Importing a vector store client into service code
• Sending twenty chunks when eight would answer better
• Trusting a citation label the model produced
• Using retrieval for counting, listing, or filtering questions
• Tuning parameters without a labelled query set
• Changing three things at once and reporting the improvement