Lesson 1: Build the eval before the feature
28 min read·13 Oct 2026
Module 10 · Lesson 1 · About 55 minutes · Needs: Module 4 Lesson 2 (the frozen split,run_prompt,score,load_split); Module 6 Lesson 1 helps (DraftReply)
The situation
It is Monday 3 August 2026, week 10. Shadow mode starts today: the assistant drafts every reply, and Lena or Ravi reviews each before it is sent.
Maya asks: "How will we know the drafts are good, other than reading a few?" Module 4 scored triage labels; a draft is open text. This lesson builds the eval for drafts before anyone tunes the drafter. The running example is T-1004: Yuki Sato bought the annual Team plan 20 days ago and wants her money back. The 14-day window has passed, so a good draft explains the rule and promises nothing.