CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 11: Inference and Serving · part 85 of 111
Part 85 · Module 11: Inference and Serving

Lab: Serve base locally, one cache, tested

71 min read·13 Oct 2026
Module 11 · Lab · About 75 minutes · Needs: lessons 1 to 7 of this module, and the Module 10 lab

What you are building

TINYFORMER-LAB AFTER MODULE 11 generate.py one cache, decoders serve/ pages, batches, HTTP compress.py quantise, prune, distil model.py: Attention takes an optional cache v0.10.0 package, otherwise unchanged test_m11_serving 10 tests m11_serving run cache, speed, memory, size git tag v0.11.0
The lab adds three parts (blue) and one small edit to attention (amber) on the unchanged v0.10.0 package (grey), pins them with ten tests (green) and tags v0.11.0.

Sol 436 at Aurora Station, the last sol of the outage. Seven lessons answered Tariq Haddad and Hana Aoki in notebook cells, and those cells grew two cache APIs and two functions called serve.

Today they become one design in tinyformer-lab: generate.py with a single cache API, a serve/ sub-package, and compress.py. Ten tests pin them, one script reruns the module's main table, and the tag is v0.11.0.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside