CourseLarge Language Models: The Complete Course · Module 10: Evaluation · part 74 of 103
Part 74 · Module 10: Evaluation

Lab: An eval suite that gates every change

57 min read·13 Oct 2026
Module 10 · Lab · About 75 minutes · Needs: lessons 1 to 5 of this module, and the Module 1, 2, 4 and 6 labs (or the cells in Step 1)

What you are building

EVALS/: ONE PACKAGE, TWO COMMANDS cases.jsonl golden-v2, 33 evals.run one record each drafter model, template evals.gate six rules PASS pin baseline labels + calibrated judge people first
The golden set (grey) runs through a drafter (amber) and the gate (blue); only a pass (green) becomes the new baseline, and human labels calibrate the judge.

It is Friday 14 August 2026, week 11. The module built an eval in notebook cells: a golden set and checks (Lesson 1), intervals and paired tests (Lesson 2), human ratings (Lesson 3), a calibrated judge (Lesson 4) and a gate (Lesson 5). This lab moves it into one package, evals/, behind the two commands Lesson 5's workflow calls: python -m evals.run, then python -m evals.gate. The running example is T-1004, Yuki Sato's refund request 20 days into an annual plan. Fifty-four calls go to the hosted model, in Steps 6 and 7, all through the course's guard.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside