Lab: One claim, measured, costed and written up

Module 15 · Lab · About 75 minutes · Needs: Modules 1 to 14 (any later module's lab can stand in: Step 1 rebuilds every file)

What you are building

On 28 May 2024 Andrej Karpathy posted "Reproducing GPT-2 (124M) in llm.c in 90 minutes for $20" (karpathy/llm.c discussion #481). It names the machine (one 8X A100 80GB SXM node, "~$14/hr" on Lambda), the time, the cost and a comparison: "We get to 29.9 here, which surpasses GPT-2 (124M) at 29.4". The next sentence is the caveat: "here we trained for 10B tokens, while GPT-3 models were all trained for 300B tokens." A configuration, a cost, a baseline and a limit, in one post.

Sol 460 at Aurora Station. Your report needs the same parts. Kofi Mensah's rule still holds: no comparison without a seed noise floor and a test. Hana Aoki asks what every result cost, the runs you threw away included. This lab builds tinyformer/capstone.py for both, runs one track and tags v1.0.0.