Lesson 3: How big the index, the GPUs and the stores must be
31 min read·
Module 3 · Lesson 3 · About 50 minutes · Needs: lesson 1 (peak RPS, requests in flight) and lesson 2 (tokens per request). No key and no GPU
The situation
Week 3. Funmi has three questions before Helen signs anything. How much memory does the vector index need for all 1,100,000 documents? If Northdesk served the small model itself, how many GPUs would Joel's target need? And how much disk do conversations and traces fill at 1 million Ask questions a day?
The running example is one Ask question at Joel's target, 2,000 tokens in and 500 out, answered from the whole corpus. Part one works every number by hand; part two runs the code that checks them.