Lesson 6: Diagnosis

Separating retrieval failure from generation failure

The wrong focaccia answer at the start of this module could have three causes, and the fixes are entirely different.

Retrieval failure. The chunk containing the answer was not returned. No prompt change can fix this, and no better model can.

Assembly failure. The chunk was retrieved but was cut off, buried in the middle of twenty chunks, or crowded out by duplicates.

Generation failure. The chunk was present and correct and the model still answered wrongly, by misreading it, by ignoring it in favour of its own knowledge, or by combining two chunks incorrectly.

Diagnose before changing anything. The procedure takes minutes and is the highest-value habit in this module.

@dataclass
class Diagnosis:
    query: str
    answer_correct: bool
    gold_chunk_retrieved: bool
    gold_chunk_rank: int | None
    gold_chunk_in_final_context: bool


def diagnose(query: str, gold_chunk_id: str, result: AnswerResult) -> Diagnosis:
    """Locate which stage failed for one query with a known correct chunk."""
    retrieved_ids = [c.chunk.id for c in result.retrieved]
    final_ids = [c.chunk.id for c in result.context_chunks]

    return Diagnosis(
        query=query,
        answer_correct=result.is_correct,
        gold_chunk_retrieved=gold_chunk_id in retrieved_ids,
        gold_chunk_rank=(
            retrieved_ids.index(gold_chunk_id) + 1 if gold_chunk_id in retrieved_ids else None
        ),
        gold_chunk_in_final_context=gold_chunk_id in final_ids,
    )

Read the result as a decision tree. Not retrieved at all means a retrieval problem, so look at chunking, the embedding model, or hybrid search. Retrieved but not in the final context means an assembly problem, so look at reranking, top-k, or deduplication. In the final context and still wrong means a generation problem, so look at the prompt, the ordering, or the model.

This requires logging the intermediate state. If your system logs only the query and the answer, you cannot diagnose anything and every investigation becomes guesswork. Log the retrieved chunk identifiers with scores, the chunks that survived reranking, and the final assembled set, for every query.

[IMAGE PROMPT M9-5
Purpose: Give learners a decision procedure for attributing a wrong answer to the retrieval, assembly, or generation stage.
Visual type: Decision tree flowchart with stage attribution.
Prompt: A clean educational decision tree reading top to bottom. The root box reads "answer is wrong". An arrow leads to the first decision diamond reading "was the correct chunk retrieved at all?". Its "no" branch exits left to a terminal box headed "Retrieval failure" listing three lines: "check chunking boundaries", "check embedding model fit", "add hybrid or keyword search". Its "yes" branch continues down to a second decision diamond reading "did it survive reranking and top_k?". Its "no" branch exits left to a terminal box headed "Assembly failure" listing "check reranker", "raise final top_k", "check deduplication". Its "yes" branch continues down to a third decision diamond reading "was it in the final prompt?". Its "no" branch exits left to a terminal box headed "Budget failure" listing "context truncation", "chunk ordering". Its "yes" branch leads to a terminal box headed "Generation failure" listing "check prompt grounding instruction", "check chunk ordering and position", "check model". A side note beside the root reads "requires logging retrieved ids, reranked ids, and final context ids".
Required elements: Three decision diamonds with their exact questions, four labelled terminal outcomes each with their listed checks, yes and no labels on every branch, the logging side note.
Style: Clean educational illustration, professional, uncluttered, high contrast, flat vector, diamonds for decisions and rectangles for outcomes.
Layout: Vertical main flow with terminal outcomes branching to the left.
Text labels: "answer is wrong", "was the correct chunk retrieved at all?", "did it survive reranking and top_k?", "was it in the final prompt?", "Retrieval failure", "Assembly failure", "Budget failure", "Generation failure", "check chunking boundaries", "check embedding model fit", "add hybrid or keyword search", "check reranker", "raise final top_k", "check deduplication", "context truncation", "chunk ordering", "check prompt grounding instruction", "check chunk ordering and position", "check model", "requires logging retrieved ids, reranked ids, and final context ids", "yes", "no".
Aspect ratio: 4:3
Accessibility: Label every branch yes or no in text and distinguish decisions from outcomes by shape rather than colour.
Avoid: Code screenshots, decorative icons, tiny text, logos, watermarks.
Alt text: Decision tree for diagnosing a wrong answer, branching on whether the correct chunk was retrieved, survived reranking, and reached the final prompt, attributing the failure to retrieval, assembly, budget, or generation.
END IMAGE PROMPT]

Recall@k and precision as debugging instruments

Two metrics, used as instruments rather than as scores to report.

Recall@k is the proportion of queries where a correct chunk appears in the top k results.

def recall_at_k(results: list[QueryResult], *, k: int) -> float:
    """Fraction of queries whose gold chunk appears in the top k."""
    hits = sum(1 for r in results if r.gold_chunk_id in [c.chunk.id for c in r.retrieved[:k]])
    return hits / len(results)

Recall@k sets the ceiling on everything downstream. If recall@50 is 0.70, then thirty percent of queries cannot be answered correctly no matter how good your reranker, prompt, or model is. This single number tells you whether to work on retrieval or stop working on retrieval, and it is the first thing to measure.

Precision@k is the proportion of returned chunks that are relevant. It matters less for the retrieval stage, where over-retrieving is deliberate, and more for the final context, where irrelevant chunks cost tokens and dilute attention.

Reading the pair together.

High recall@50 and low recall@5 means retrieval is finding the right chunks and ranking them badly, which is exactly the case a reranker fixes.

Low recall@50 means retrieval is genuinely missing content, and reranking cannot help because the chunk is not in the candidate set. Look at chunking and the embedding model.

High recall@5 and wrong answers means the problem is downstream, in assembly or generation.

Two related metrics worth knowing. Mean Reciprocal Rank rewards putting the correct chunk higher, which is what a reranker is optimising. NDCG handles the case where several chunks are relevant to differing degrees. Both are refinements of the same idea, and recall@k is enough to start.

A small labelled query set before tuning anything

Everything in this module is a parameter: chunk size, overlap, splitting strategy, embedding model, dimensions, fusion weight, top_k, reranker, ordering, and query transformation. Tuning them without measurement is guessing, and the changes interact, so intuition about them is unreliable.

Build fifty to a hundred labelled queries before changing anything. It takes an afternoon and it is the difference between engineering and guessing.

@dataclass(frozen=True)
class LabelledQuery:
    query: str
    gold_chunk_ids: list[str]        # chunks that genuinely answer it
    gold_answer: str | None = None   # for judging the generated answer
    category: str = "general"        # to slice results by query type

Where the queries come from. Real user queries from your logs are best, because synthetic queries are systematically easier and more grammatical than what people actually type. Failure cases you have already found should all be in the set. Include the hard categories deliberately: multi-hop questions, exact-identifier lookups, questions with no answer in the corpus, and ambiguous questions.

That fourth category matters more than it sounds. A system that confidently answers questions the corpus cannot answer is worse than one that says it does not know, and unless unanswerable queries are in your set you will never measure that behaviour.

Labelling. For each query, find the chunk or chunks that genuinely answer it. This is manual and it is the work. Doing it yourself has a side benefit: you read your own chunks, which is how most people discover their chunking is broken.

Using it.

async def evaluate(queries: list[LabelledQuery], pipeline: Pipeline) -> EvalReport:
    """Run the labelled set and report retrieval metrics by category."""
    results = [await pipeline.run(q.query) for q in queries]
    return EvalReport(
        recall_at_5=recall_at_k(results, k=5),
        recall_at_50=recall_at_k(results, k=50),
        mean_reciprocal_rank=mrr(results),
        by_category=recall_by_category(results, queries),
    )

Then change one thing at a time. Chunking strategy, then embedding model, then fusion, then top_k. Changing several at once tells you the combination is better or worse without telling you which part did it.

Two disciplines that make the numbers trustworthy. Run the same configuration twice and see how much the number moves, since anything smaller than that variation is not a result. And slice by category, because an average hides that your recall on exact-identifier queries is 0.4 while your overall number looks acceptable, and that slice is exactly what hybrid search would fix.

Concept check. Your recall@50 is 0.92 and recall@5 is 0.55. Answer quality is poor. What do you do, and what would be a waste of time?

Answer

Add or improve a reranker. Retrieval is finding the correct chunk for 92 percent of queries, so the candidate set is good, and the ranking is putting it outside the top five more than a third of the time. That gap between recall@50 and recall@5 is precisely the problem a cross-encoder reranker solves, and this is the clearest signal you will get that reranking will pay off.

A waste of time: changing the embedding model, re-chunking the corpus, or adding hybrid search. All of those improve which chunks enter the candidate set, and your candidate set is already right 92 percent of the time. You would spend hours re-embedding two million documents to fix a problem you do not have.

Also worth checking once the reranker is in: whether your final top_k is large enough, and whether duplicates are consuming slots. Both affect the same gap and cost nothing to test.