CourseAI / ML Engineer: The Complete Course · Module 12: Foundation Models · part 74 of 93
Part 74 · Module 12: Foundation Models

Lesson 3: Evaluating foundation models for Kestrel's own task

34 min read·
Module 12 · Lesson 3 · About 55 minutes · Needs: lessons 1 and 2, and course-guide/02 (the setup cell and ask()). Runs on a CPU; the judge's hosted path needs a key and is optional.

The situation

Week 12 at Kestrel, day five. There are now three candidate message models: the base SmolLM2, lesson 1's adapter (v1), and a v2 that Adaeze's team trained on all 960 messages, disputed labels included, because "more data is better". Adaeze wants one rule for which version ships, and Hannah wants to know whether an LLM can grade reply drafts so her team does not have to.

The running example is M0001 (simulated), "hiya you said tomorrow 4 my parcel. it is now late! Regards", now one case in a fixed eval set.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
Lesson 3 builds the evidence and the gate that decide which message model version reaches the registry (blue).

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan