Lesson 1: Attention, the step where words help each other

Module 6 · Lesson 1 · About 35 minutes · Needs: Module 4 (what an embedding is)

The situation

Priya forwards a complaint. A customer wrote "My vacuum dock arrived cracked, please replace it" and BrightCart's assistant replied with instructions for replacing the vacuum. The customer wanted a new dock.

A person reads that sentence and knows "it" means the dock without thinking. A model has to work it out. The mechanism it uses is called attention (in full, self-attention: the words of one sentence attending to each other), and it is the one idea that made modern language models possible. This lesson shows how it works, one picture at a time, using that sentence.