Lesson 5: Regression gates, user signals and safe online tests
32 min read·13 Oct 2026
Module 11 · Lesson 5 · About 60 minutes · Needs: Module 4 Lesson 4 (hit rate and MRR), Module 5 Lesson 3 (permission filters) and Module 9 Lesson 1 (the answer contract)
The situation
On Friday 9 October 2026 Grace sets the bar for the agents' pilot on 2 November: a regression gate must pass, and keep passing as documents and code change. Kwame wants to spend the week on a bigger embedding model.
You open the prototype's query log instead. One entry, "cracked jar covered?" (1 October), is this lesson's running example: you will read it, name its failure, turn it into a test and watch the gate use it. The Hollin data is simulated.