Lesson 4: Embeddings, the output head and the shape of a model
27 min read·13 Oct 2026
Module 3 · Lesson 4 · About 50 minutes · Needs: Module 3, Lessons 1 and 2 (ModelConfig,count_params,FeedForward,Block); Module 2, Lesson 5 (GroupedQueryAttention)
The situation
Sol 407 at Aurora Station. Hana Aoki, the Compute Officer, asks: "How much of a model is just the token table? What would a published one cost to hold and to train?" Kofi Mensah adds: "base ties its head to its embedding table. What does that save, and cost?"
This lesson builds both ends of the model and TinyFormer, which joins them to Lesson 2's blocks, for tinyformer/model.py. The running example is line 4 of Ana Dubois's log for sol 286: "Ana noted that pressure held stead", where the model must predict "y".