Lesson 3: Sharding, tensor and pipeline parallelism

Module 9 · Lesson 3 · About 50 minutes · Needs: Lesson 2 (data parallelism and the two-process launcher); course-guide/06, blocks 15 to 18 (the reference model)

The situation

Sol 426 at Aurora Station. Hana Aoki, the Compute Officer, is costing a model 1,000 times bigger than base on hired GPUs on Earth. Lesson 2 copied the whole model onto every device. Hana's next question is blunt: "When does a copy stop fitting, and what do people do then?"

The running example is that pair of models: base, 800,512 parameters, which you can split for real on two CPU processes, and its 1,000-times cousin, about 800 million parameters, which you cost with arithmetic. Lesson 4 fixes that cousin as big (805,777,408 parameters); the difference moves no conclusion here (12.8 GB of state becomes 12.9 GB). By the end you can tell Hana which kind of splitting her model needs, and prove each kind gives the same answer as no splitting.