Lesson 3: Speculative decoding and continuous batching, measured

Module 11 · Lesson 3 · About 55 minutes · Needs: Module 2, Lesson 5 (the KV cache idea); the base recipe (course-guide/06, block 19); Module 8's package

The situation

Sol 433 at Aurora Station. The link to Earth is down for three sols, so every crew question goes to the station's own model, base, on small CPU servers. Hana Aoki, the Compute Officer, has two complaints: a reply comes out one character at a time, and when sixteen people ask at once, short questions wait behind long ones.

Her rule: "Faster is fine. A different answer is not." The running example is one maintenance log header, "Log 5107, sol 433, the night shift.", continued for 80 characters.