Lesson 2: Positions inside attention: relative, ALiBi and rotary

Module 4 · Lesson 2 · About 55 minutes · Needs: Module 4, Lesson 1 (the position switch, AbsolutePositions, PosLab); Module 2, Lesson 5 (grouped-query attention)

The situation

Lesson 1 left Kofi Mensah a red row: past the trained length, both absolute methods failed. Yet base uses rotary positions, a choice nobody at Aurora Station has measured. "Earn it," Kofi says. "Build the methods that act inside attention, check rotary's claim in code, and race them all, three seeds each."

This lesson adds relative positions, ALiBi and rotary embeddings to tinyformer/positions.py, behind the same position switch.