Lesson 4: State-space models, hybrids, and what attention keeps

Module 14 · Lesson 4 · About 55 minutes · Needs: Module 10, Lesson 2 (linear attention and its recall gap); Lesson 1 of this module (the induction shortcut); shared blocks 1, 9 and 15 to 18

The situation

Sol 452 at Aurora Station. Tariq Haddad, the Comms Officer, has read about Mamba: no attention, no key/value cache, and memory that stays the same size however long the message. base reads only 128 characters at a time, and the station's CPUs are short of memory. Should the lab swap attention out?

Kofi Mensah sets the terms: same size, same steps, three seeds, and a recall test with no shortcut. The running example is one held-out row of it: eight crew, each paired with a system, then all eight asked again.