CourseAI / ML Engineer: The Complete Course · Module 5: Data Engineering for ML · part 36 of 93
Part 36 · Module 5: Data Engineering for ML

Lab: The daily data path, with a contract

88 min read·
Module 5 · Lab · About 60 minutes · Needs: lessons 1 to 4 of this module, and the Module 4 lab (or nothing: Step 1 rebuilds Modules 1 to 4)

What you are building

Kestrel's daily data path Nothing reaches the lake unchecked Active flow Background or async 1 Processing step number Depots 9 scan files a night extract Lesson 1 Landing Parquet landing/ as sent Contract Pandera fix grams 1.1 check 18 columns Clean side lake by date DuckDB reads online.db route rates lineage.jsonl 1 2 3 4
Each night the depots' files land as sent (gold), the contract fixes the agreed unit and checks every rule (teal), and only a passing day reaches the lake and the online store, with lineage recorded (purple); circles give the order.

Tomasz's question is the fourth of the course's five: "What happens if the feature pipeline silently changes a column?" On 7 September it did: scanner software v4 sent weight_kg in grams, and nobody told the ML team. The lessons built every piece of the answer. This lab joins them into one daily run that nothing can skip.

pipelines/daily.py takes one ship date through four steps: ingest each depot's scan file into the landing area, check the day against the data contract, write the clean day into the lake, and build the route features for the online store, recording lineage as it goes. The lab also adds tests/test_contracts.py, the contract written for people in docs/data-contract.md, and docs/adr/ADR-003-data-path.md. No cell calls a hosted model.

Open a new Colab notebook, call it module-05-lab.ipynb, and run each cell in order.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan