Lesson 2: The scaling factor, derived, measured and removed

Module 2 · Lesson 2 · About 45 minutes · Needs: Module 2, Lesson 1 (naive_attention, the Maya trace); Module 1, Lesson 2 (the copy race)

The situation

Sol 404 at Aurora Station. Kofi Mensah is reading tinyformer/attention.py line by line. He stops at scores / math.sqrt(q.shape[-1]). "Every line in this lab earns its place with a test. What does this one do, and what breaks without it?"

This lesson answers with a derivation checked in code, measurements on random and trained heads, and a training run without the line. If you met the factor in the ai-engineering course (Module 6, Lesson 1), here you test it.