CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 2: Self-Attention Mechanics · part 16 of 111
Part 16 · Module 2: Self-Attention Mechanics

Lesson 4: Masking, built, tested and broken on purpose

19 min read·13 Oct 2026
Module 2 · Lesson 4 · About 50 minutes · Needs: Module 2, Lessons 1 and 2 (naive_attention, its mask argument, the copy task)

The situation

Sol 405 at Aurora Station. naive_attention has had a mask argument since Lesson 1, but the lab has only ever passed the causal triangle. Kofi Mensah sets a rule: "A name's outputs must not depend on its neighbours in a batch, or on what comes after it. Test each mask, and show me what leaks when one is wrong."

The running example is two crew names in the shared character tokeniser: Maya (ids 32, 45, 68, 45) and Tariq (39, 45, 61, 53, 60). You will batch, pad and pack them, and measure every leak.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.