Lesson 2: Boxes, masked patches and CLIP on test cards
21 min read·13 Oct 2026
Module 13 · Lesson 2 · About 60 minutes · Needs: Lesson 1 (patches as tokens, a small ViT); Module 7, Lesson 1 (the encoder's full mask); Module 8, Lesson 2 (objectives as what is read and what is scored)
The situation
Sol 447 at Aurora Station. Lesson 1's vision transformer names the shape on a camera test card. Isla Duarte wants more: where each shape is, so a robot arm can reach it; an encoder that learns from unlabelled cards; and classes she can ask for in words.
The running example is test card 1601, "three green triangles", from the held-out test set.