CourseAI / ML Engineer: The Complete Course · Module 12: Foundation Models · part 75 of 93
Part 75 · Module 12: Foundation Models

Lab: Message models evaluated, gated and costed

186 min read·
Module 12 · Lab · About 60 minutes · Needs: lessons 1 to 3 of this module, and the Module 11 lab (or nothing: Step 1 rebuilds Modules 1 to 11). Runs on a CPU; downloads SmolLM2-135M-Instruct (257 MiB)

What you are building

One harness, three kinds of model Same cases, same gate, then a decision Active flow Background or async 1 Processing step number Optional or fallback Candidates Module 4 transformer SmolLM2 + LoRA artifacts/intent_lora hosted model via ask(), fakes here evals/ messages_eval.py 246 fixed cases regression.py slice by slice Verdict PROMOTE BLOCK REFUSE Decision ADR-009 self-host 1 2 3 4
The Module 4 transformer, the LoRA adapter and a hosted model (dashed: scripted fakes here) run on the same 246 cases; the regression gate promotes, blocks or refuses; ADR-009 records where message models run; circles give the order.

Hannah's team routes about 4,000 customer messages a day by hand (illustrative, Lesson 2). Three kinds of model could do it: Module 4's small transformer, Lesson 1's LoRA adapter on SmolLM2, or a hosted model behind an API. This lab puts all three through one harness: evals/messages_eval.py runs any of them on the same 246 cases, evals/regression.py decides promote, block or refuse, and kestrel/llm.py holds the course's setup cell and ask() guard so a hosted model is one setting away. ADR-009 then records whether Kestrel self-hosts or calls an API.

The owner's rule for this course: no hosted calls. The hosted path runs here with scripted fakes through ask(), objects with a model's invoke method and fixed, labelled behaviour. Their scores test the harness; they are not results from any model.

Open a new Colab notebook, call it module-12-lab.ipynb, and run each cell in order.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan