Lesson 1: The training loop, and what the loss is telling you

Module 6 · Lesson 1 · About 50 minutes · Needs: Module 5 (the model, its tests, its numerics); Module 1, Lesson 1 (softmax)

The situation

Sol 412 at Aurora Station. base passes Kofi Mensah's tests and trains to a validation loss of 0.256. Kofi asks: "0.256 against what? What would a model that learnt nothing score? And which loop made it? I want one recipe, in one file, for every later run."

This lesson writes that loop, tinyformer/train.py, and fixes the base recipe in it: 600 steps, AdamW, warm-up then cosine. It derives the loss, sets floors under it and reads base's curve. The running example: the model has read "Bruno Hoffman works in th" and must predict "e".