Module 1: The Road to Attention
1 min read·13 Oct 2026
Why recurrent networks gave way to attention, measured rather than asserted. Everything runs on a Colab CPU.
What you need first
- Python and NumPy; matrix products, transposes and the chain rule
- Having trained one neural network (AI Engineering, Module 4, if not)
- If you have done AI Engineering Module 5 (why recurrent models struggle) and Module 6 (attention in overview), skim Lesson 2's section 4 and Lesson 3's section 6
- No GPU, no API key
What you will build
The tinyformer-lab repository, with the shared data as a tested package and a three-seed race of a GRU, a GRU with attention and a one-layer transformer encoder, tagged v0.1.0.
Lessons
| File | Title | Minutes | Depth tags |
|---|---|---|---|
01-prerequisites-as-experiments.md | Lesson 1: Six prerequisites, each checked by running code | 45 | Essential |
02-why-recurrence-needed-replacing.md | Lesson 2: Why recurrence needed replacing, measured | 45 | Essential, Recommended |
03-attention-before-transformers.md | Lesson 3: Attention before transformers, rebuilt on one task | 50 | Essential, Recommended |
Extras
04-module-lab.md: Lab: Start tinyformer-lab with one honest race (50 minutes). Nine seeded runs read against a noise floor.05-incident-file.md: Incident: The causal flag that masked nothing (30 minutes). PyTorch issue #96941 (2023), whereis_causal=Truewithout a mask did not mask, replayed on the lab's encoder and caught with a causal check.