Lesson 1: The data pipeline, and comparing objectives fairly

Module 8 · Lesson 1 · About 50 minutes · Needs: Module 6, Lessons 1 and 4 (the proxy, train with loss_fn, data and monitor, the test split)

The situation

Sol 421 at Aurora Station. Lesson 2 will race eight pretraining objectives, and Kofi Mensah has set the terms first: "Before any race, show me the corpus, and show me the race is fair." He wants a corpus report, a trained tokeniser, and a comparison harness with seeds.

This lesson builds the pipeline from documents to batches, then the harness. The running example is document 0, the assignment sheet for sol 379, and its first row: "Bruno Ferreira works in the water plant on the coolant loop."