Lesson 3: What happens inside the model server
28 min read·
Module 7 · Lesson 3 · About 50 minutes · Needs: Module 3 lessons 3 and 4 (GPU memory, KV cache, break-even). No key and no GPU
The situation
Week 7. Module 3 sized gpt-oss-20b at 4 H100s for Joel's target and found self-hosting cheaper than Groq only before people costs. Helen asks Funmi: "Which knobs inside a model server move the number of GPUs, and do we want to own them?"
The running example is one Ask question at the target, 2,000 tokens in and 500 out, one of 463 requests in flight at peak. Part one explains what a server does with it, by the numbers; part two measures it on a small open model on CPU.