Lesson 2: Build the test set before you tune a prompt
23 min read·13 Oct 2026
Module 4 · Lesson 2 · About 50 minutes · Needs: Module 1 Lesson 1 (the setup cell, the tickets, the first triage prompt); Module 1 Lesson 4 helps (a score with an interval)
The situation
It is Monday 22 June 2026, week 4. You are about to improve the triage prompt, and the temptation is obvious: change a line, try T-1001, Ana Duarte's duplicate charge of 288 USD, see billing high, and call it better.
Maya stops you: "Better on which tickets, measured how?" Before any prompt changes, this lesson builds what judges prompts: the 36 tickets Lena and Ravi labelled on 8 June, a split that cannot change unnoticed, a control prompt, and a harness. T-1001 is the running example.