Lesson 3: Connecting pixels, frames and sound to a language model

Module 13 · Lesson 3 · About 70 minutes · Needs: the shared data and base recipe (course-guide/06, blocks 1 to 8, 10 and 12 to 19); Module 4, Lesson 4 (patches); Module 12, Lesson 1 (loss masking). Lessons 1 and 2 of this module help but are not required

The situation

Sol 447 at Aurora Station. Isla Duarte, the robotics engineer, wants the robots to say in words what is on a camera test card. base writes station text, but it cannot see.

This lesson gives it eyes: an image encoder, a small connector that turns image features into vectors base can read, and a training recipe. Then it turns video and sound into tokens. The running example is held-out card 1602: three yellow circles.