CourseAI / ML Engineer: The Complete Course · Module 5: Data Engineering for ML · part 34 of 93
Part 34 · Module 5: Data Engineering for ML

Lesson 3: Data you can trust: contracts, versions and labels

30 min read·
Module 5 · Lesson 3 · About 55 minutes · Needs: lessons 1 and 2 (the nightly files and the lake), pandas, and Git basics (commit, checkout).

The situation

Week 5, Thursday. Tomasz mentions, in passing, that scanner software v4 went to every depot on 7 September. Nobody told the ML team what else changed. Adaeze asks for three things before any model trains on September: a written agreement on what the scan feed contains, a way to get back any past version of the data, and proof that the customer-message labels and the personal data in them are handled properly. Hannah owns that last part.

One depot-day runs through the lesson again: Dartford on Monday 7 September 2026, 435 parcels (simulated). Every one of their weights is wrong by a factor of a thousand, and nothing has crashed.

WHERE THIS FITS The Kestrel system this lesson SHARED SERVICES Depot scans parcels, routes Ingestion nightly batch Validation Pandera checks Data lake Parquet, DuckDB Experiments MLflow runs Training LightGBM, Optuna Online store Feast, Redis Features point in time Gates vs the champion Registry staged versions Serving API FastAPI, ONNX Consumers vans, customers Retraining trigger, budget Alerting on call Monitoring drift, Evidently Late labels a day later Orchestrator Prefect flows Versions DVC, Git CI/CD GitHub Actions Containers Docker images
This lesson adds the data contract that validates every batch, and the versions of the data in the lake (blue); the rest of Kestrel is grey for now.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan