Inference and Serving roadmap

Running a model fast and cheaply enough to be a product. Memory arithmetic, quantisation, batching, and the scheduling decisions that determine whether your GPU is doing useful work or waiting.

Deep dive11 stages154 topics

All roadmaps
0 of 154 done
Essential, don't skip Recommended, skip only with a reason Optional, awareness is enough Course on PracticAI

This is about serving models yourself. If you call a hosted API, sections 7 and 10 still apply and the rest is background you can read once. Section 2 is the arithmetic everything else rests on - do it by hand at least once, because a serving plan that fails the memory calculation fails regardless of how good the rest of the design is.

1

How Inference Works

          2

          Memory Arithmetic

          The calculation everything else depends on. Do it by hand once and the rest of this roadmap makes sense.

                      3

                      Quantisation

                              4

                              Attention and Caching

                                      5

                                      Batching and Scheduling

                                      The single largest throughput lever, and the one that determines whether your GPU is busy or idle.

                                              6

                                              Serving Stacks

                                                      7

                                                      Latency Engineering

                                                      Applies whether you serve the model or call an API. Perceived latency is a product property, and it is not the same as measured latency.

                                                                  8

                                                                  Throughput and Scale

                                                                          9

                                                                          Benchmarking

                                                                          Vendor numbers describe a workload that is not yours. Benchmark honestly or you will size the deployment wrong.

                                                                                  10

                                                                                  Cost

                                                                                          11

                                                                                          Operations

                                                                                                  Sizing a deployment

                                                                                                  Work these in order. Most serving disappointments come from skipping to step five and discovering the arithmetic never worked.

                                                                                                  1. Weights. Parameters times bytes per parameter. A 7B model at FP16 is roughly 14GB before anything else exists.
                                                                                                  2. KV cache. Scales with batch size times sequence length. At long contexts and real concurrency this frequently exceeds the weights, and it is the number people forget.
                                                                                                  3. Overhead. Activations, fragmentation, framework overhead. Leave real headroom rather than a token allowance.
                                                                                                  4. Quantise if it doesn't fit - and measure the quality cost rather than assuming it's negligible.
                                                                                                  5. Now choose hardware , knowing whether you are memory-bandwidth-bound or compute-bound.
                                                                                                  6. Benchmark under realistic load. Your prompt lengths, your output lengths, your concurrency. Vendor numbers describe a different workload than yours.
                                                                                                  7. Compare against the API honestly. Include idle GPU time, on-call, and the engineering months. Self-hosting usually wins on volume and loses on everything else.

                                                                                                  Time to first token is dominated by prefill and scales with input length. Time between tokens is dominated by memory bandwidth. They are different problems with different fixes, and conflating them wastes weeks.

                                                                                                  A GPU sitting idle between requests costs the same as one doing work. Most of the engineering here is not making the model faster - it is making sure the hardware you are already paying for is busy.