Lesson 5: Head variants and what attention really costs
23 min read·13 Oct 2026
Module 2 · Lesson 5 · About 55 minutes · Needs: Module 2, Lessons 1 to 3 (naive_attention,MultiHeadAttention, loading GPT-2); shared blocks 1 and 15 to 17
The situation
Sol 406 at Aurora Station. Tariq wants base to read whole transmissions from Earth, 4,096 characters each, not 128. Hana Aoki, the Compute Officer, owns the small CPU servers that would run it. "What does attention cost on my machines at that length, in memory and in time? And Lesson 3 ended by saying modern models keep fewer key and value heads. Should we?"
This lesson measures both on the base shape (4 layers, 4 heads of 32, width 128) and adds grouped heads, cross-attention and a sliding window to tinyformer/attention.py.