Lesson 2: Two attention paths, starting weights and reference checks

Module 5 · Lesson 2 · About 55 minutes · Needs: Module 3's lab (ModelConfig, Block, TinyFormer at v0.3.0); Module 2, Lessons 1 and 4 (attention, masks)

The situation

Sol 409 at Aurora Station. Kofi Mensah is still reviewing tinyformer/model.py, with three questions. "If you switch attention to PyTorch's fused kernel, how do you know nothing changed? Every weight starts at 0.02: why? And you wrote every test yourself. What says the model computes what a transformer computes?"

This lesson answers all three on one running example, row 0 of the copy task, and ends by loading GPT-2 small into TinyFormer and matching its logits.