Lesson 5: Measuring whether the answers are right

Module 8 · Lesson 5 · About 40 minutes · Needs: Module 5 lab (hit@1 and hit@3), Lessons 2 to 4 of this module (embeddings, retrieval, grounded answers)

The situation

The assistant now answers from the help centre with citations (Lesson 4). Priya asks: "Is it right, and will you know when it stops being right?" Dev wants to swap the retriever next week and asks how he will know if the swap made things worse.

Reading answers by eye does not scale, and it does not say where a wrong answer went wrong. This lesson builds a small test set, scores each stage separately and labels every failure with its stage. The running example is a shared ticket: "I returned my speaker last week. When do I get my money?"