CourseAI / ML Engineer: The Complete Course · Module 9: Serving and Inference · part 58 of 93
Part 58 · Module 9: Serving and Inference

Lab: The scoring service, batch and online

132 min read·
Module 9 · Lab · About 60 minutes · Needs: lessons 1 and 2 of this module, and the Module 8 lab (or nothing: Step 1 rebuilds Modules 1 to 8)

What you are building

The scoring service, batch and online One adapter, one model file, two paths Active flow Background or async 1 Processing step number Optional or fallback Callers depot scanner one parcel 06:00 job every parcel due service/main.py FastAPI gate 429 when full Served grams fix, dates fallback ladder cache, then logit Models ONNX Runtime model.onnx v2 fallback.joblib logistic Stores Redis cache scores.db 06:00 run 1 2 3
A scanner's request passes the gate (green panel) and the adapter to model v2 in ONNX (blue); a failure falls back to the cache and the logistic model (dashed); the 06:00 job uses the same adapter and writes scores.db (gold); circles give the order.

Kestrel scores each parcel twice: online, one at a time, when a depot scanner collects it, and in a batch at 06:00 on its promised day, so the warnings go out before the vans leave. The lessons built every piece of that service around stand-in models trained in the notebook. This lab assembles the service Kestrel would deploy, around the real thing: model v2 from artifacts/model.onnx (Module 8 lab), with Module 3's logistic regression as the fallback.

The lab adds service/main.py (the adapter that feeds raw scanner rows to Kestrel's models, the app and the factory a server starts), tests/test_service.py, docs/slo.md, docs/adr/ADR-006-serving-shape.md, and the files that run it outside Colab: service/Dockerfile and docker-compose.yml with the API, Redis and MLflow, written and given as local steps, not run here. No cell calls a hosted model.

Open a new Colab notebook, call it module-09-lab.ipynb, and run each cell in order.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan