Lesson 1: What is expensive, and sparse attention

Module 10 · Lesson 1 · About 55 minutes · Needs: Module 2, Lesson 5 (attention FLOPs, window_mask); shared blocks 1 to 9 and 15 to 19

The situation

Sol 431 at Aurora Station. Tariq Haddad, the Comms Officer, reports that long transmissions from Earth are being split. base reads 128 characters at a time, so a 4,096-character transmission becomes 32 pieces, and a question in piece 30 cannot see the answer in piece 2.

"Can it read the whole thing at once?" This lesson finds what is expensive in base, measures where attention takes over, and tests the oldest way out: let each token read only some of the others. The running example is that one transmission.