CourseAI Engineering: The Complete Course · Module 12: Evaluation and Observability · part 91 of 105
Part 91 · Module 12: Evaluation and Observability

Lesson 2: Seeing what the answer engine did, request by request

22 min read·13 Oct 2026
Module 12 · Lesson 2 · About 55 minutes · Needs: Module 1 Lesson 3 (JSON logs), Module 7 Lesson 5 (cost per request), Module 8 Lesson 4 (the citation check)

The situation

The answer engine is live. Priya forwards two complaints: one customer waited "ages", another got a reply citing the wrong article. Marta asks what the week cost. Dev asks whether the failures are one bug or several.

Lesson 1 tested the engine before release; this lesson watches it after. You will run the forty labelled tickets as one simulated week, Monday 5 to Friday 9 October, and trace ticket 4, "When will my refund show up on my card?", end to end. A fake model with labelled simulated timings keeps the numbers repeatable; two calls are real.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.