Lesson 3: Building an evaluation set people actually ask
34 min read·13 Oct 2026
Module 11 · Lesson 3 · About 60 minutes · Needs: recall and hit rate in plain words (Module 4), the shared questions and query log (course-guide/06); no code from Lessons 1 and 2
The situation
Grace's pilot for her support agents starts on 2 November 2026 if the Module 11 gate passes, and a gate is only as good as its questions. Hollin Answers has 36 labelled questions, written by the course team. Grace's agents ask differently.
On 1 October 2026 an agent typed "cracked jar covered?" into Kwame's prototype, got the archived 2024 terms, gave a thumbs down and escalated. No labelled question looks like that. This lesson follows that query from the log into the evaluation set.