CourseRetrieval-Augmented Generation: The Complete Course · Module 11: Evaluation · part 79 of 95
Part 79 · Module 11: Evaluation

Lab: The evaluation harness and its gate

68 min read·13 Oct 2026
Module 11 · Lab · About 90 minutes · Needs: lessons 1 to 5 of this module, and the Module 6 and Module 9 labs (or the cells in Step 1)

What you are building

EVALS/: FOUR FILES, ONE COMMAND questions.py 52 cases harness.py metrics, runs judge.py nightly only gate.py --tier exit 0 exit 1
The lab's four files (blue): the set feeds the harness, the judge (amber, a model) grades answers at night, and the gate turns the comparison into exit 0 (green) or exit 1 (red).

On Friday 9 October Grace's pilot is three weeks away, and it starts only if a gate passes. The five lessons each built a piece of that gate in its own notebook, with its own copy of the questions and, in Lesson 5, a stand-in search. This lab puts them into evals/, the folder the course guide names. questions.py holds the one evaluation set. harness.py holds the metrics and the two runs: retrieval on Module 6's real search, and answers through Module 9's respond(). judge.py is Lesson 4's judge. gate.py is what Lesson 5's workflow calls: python -m evals.gate --tier pr or --tier nightly, which exits with 1 when something got worse. The model is called in Steps 5 and 7 only.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside