Lesson 1: Tests that say if a change helped
21 min read·13 Oct 2026
Module 12 · Lesson 1 · About 45 minutes · Needs: Module 7 (the answer engine, structured replies), Module 8, Lesson 5 (retrieval metrics and groundedness)
The situation
Dev wants to shorten the answer engine's prompt by one line. Priya asks: "Will it still tell Tom his return window has closed?" Nobody can say; the only test so far is reading a few replies by eye.
Module 8 scored retrieval for the help-centre answers. This lesson scores the whole engine end to end: an eval set from real messages, a harness, and fair comparisons of prompts, judges and releases. The running example is Tom Becker's logged message: "Please refund order 7120, the bulbs are the wrong fitting."