Lab: An eval harness with traces and a gate

Module 12 · Lab · About 60 minutes · Needs: lessons 1 and 2 of this module (evaluation, observability)

What you are building

Dev still wants to shorten the answer engine's prompt, and Priya still wants to know whether Tom will hear about the 30-day window. This lab builds the piece Module 12 adds to BrightCart: tests that tell you whether a change made the assistant better or worse. A harness runs the engine over a 10-case eval set saved as JSON, scores key facts and groundedness by word overlap, and saves each run under its prompt version. Every model call is traced with Lesson 2's tracer, so each score links to a span tree with latency, tokens and cost. Two prompt versions are compared case by case, and a gate script fails the release when correctness drops. One run makes 21 hosted calls and no judge calls.