CourseTransformers: From Attention to Trained, Served and Interpreted Models · Module 6: Training Dynamics · part 48 of 111
Part 48 · Module 6: Training Dynamics

Incident: A 104B model that was really 58B

16 min read·13 Oct 2026
Module 6 · Incident file · About 40 minutes

What happened

TR8-104B CHRONICLE, 28 SEP TO 18 OCT 2021 Exp 1 spike at 7,000 Exp 2 new seed Exp 3 stabler attention Exp 4 beta2 0.95 Exp 5 from step 0 Exp 6 FFN was 20480 Exp 7 to 10 shape, LR Exp 11 init std 0.006 red: failed; amber: the setup error found; green: trained
Five experiments failed (red) before Experiment 6 found the feed-forward width was the 13B model's (amber); the instability outlasted the fix until Experiment 11 (green).

In September and October 2021 the BigScience workshop tried to train a 104-billion-parameter GPT-style model, tr8-104B, as a step towards its large multilingual model. Stas Bekman kept a public log of the attempt, "tr8-104B Chronicles", in the BigScience repository (github.com/bigscience-workshop/bigscience, first entry 28 September 2021). Its stated focus: "any encountered problems and their diagnosis and solutions/prevention."

Experiment 1 ran on 64 nodes. "Somewhere between iteration 7000 and 7010 lm loss jumped from 6.4 to 14", then sat near 7 "w/o any change, and later it went into NaN." The team rolled back to the checkpoint at step 6210 and changed the seed. The glitch came back "even sooner". Experiment 3 tried a more numerically stable attention; Experiment 4 rolled back to step 3000 with Adam's beta2 lowered to 0.95; Experiment 5 started again from step 0. Each went wrong.

Then, in the entry for Experiment 6 (in the log by 4 October 2021): "Discovered a really bad mistake in the setup that impacted all the previous experiments that failed. When changing from 13B to 104B I did not update FFN_HIDDEN_SIZE and it remained 20480 when it should have become 65536, so it has been a very lopsided 58B model instead of 104B all along."

The fix did not end the instability: with the corrected shape, the outcome was "very similar to Exp 4 and 5". Five experiments' worth of diagnosis had been about a model nobody meant to train.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside