CourseAI / ML Engineer: The Complete Course · Module 12: Foundation Models · part 73 of 93
Part 73 · Module 12: Foundation Models

Lesson 2: Serving a foundation model: memory, batching and cost

33 min read·
Module 12 · Lesson 2 · About 50 minutes · Needs: lesson 1 (the SmolLM2 model and its adapter), Module 8 lessons 2 and 3 (KV caching, dynamic batching, the benchmark harness). Runs on a CPU.

The situation

Week 12 at Kestrel, day three. Lesson 1 produced a 9.4 MiB adapter for message intents. Rhys must serve it and asks two things: how much memory, and what p99 under load. Imran asks the other: is a machine of our own cheaper than paying an API per token? Hannah's rule still stands: customer text stays inside Kestrel unless a decision record says otherwise.

The running example is the same message, M0001 (simulated), 17 tokens long, and an illustrative inbox of 4,000 messages a day, added in this lesson.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
Lesson 2 is about serving the adapted model to its consumers (blue); the rest of Kestrel's system is grey here.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan