CourseAI / ML Engineer: The Complete Course · Module 7: Training at Scale · part 48 of 93
Part 48 · Module 7: Training at Scale

Incident: A training run that hung without failing

8 min read·
Module 7 · Incident file · About 30 minutes · Needs: Lesson 2 (checkpoints, the Young and Daly interval, resuming) and Lesson 3 (throughput as a metric)

What happened

WHAT HAPPENED BLOOM's training, as logged 11 Mar 2022 launch 21 Mar GPU crash, 7.3 h 28 Apr 149 to 140 TFLOPs 30 Apr eval hang, 18 h 9 May hang again, 6 h 18 May hang again, 11 h
BLOOM's logbook, March to May 2022: one crash cost 7.3 hours, a slow GPU cut throughput (amber), and three hangs at evaluation cost 18, 6 and 11 hours (red).

BigScience trained BLOOM, a 176-billion-parameter multilingual model, on the Jean Zay supercomputer in France, and kept a public logbook: "tr11-176B-ml chronicles" (BigScience Workshop, GitHub, entries March to July 2022). "The training launched on March 11, 2022". A week later it ran at "149-150 TFLOPs per GPU".

WHAT HAPPENED Hours lost, by kind of failure 30 Apr hang 18 h 18 May hang 11 h 21 Mar crash 7.3 h 9 May hang 6 h crash, after at most 3 h
The hangs (red) cost more than the crash (amber); once checkpoints came every 100 iterations a crash cost at most 3 hours (green), but a hang's cost had no limit.

The logbook counts its losses in hours. On 21 March a GPU crashed with a CUDA "unknown error": "we lost 7.3h of work, it failed 2 iterations before checkpoint saving time". With a 2.3 TB checkpoint taking 40 seconds to save, the team moved to saving every 100 iterations, so "we would lose at most 3h". By April: "Every week or so we have hardware issues where one of the gpus dies."

The worst losses were not crashes. On 30 April "the eval stage started at the end of iteration 45000 and got stuck". It was "stuck for at least 3 hours. No timeouts from NCCL or distributed, just gpus spinning at 100%". It was a weekend; "by the time I found the cause and rolled back we lost about 18h of training". The hang came back on 9 May ("Lost 6h of work") and 18 May ("We lost ~11h on this event"), and was traced to a DataLoader worker thread holding a lock; the workaround was num_workers=0 for validation.

The rest of this course is yours to keep

This course is bought on its own, once, and stays readable afterwards, including the parts added to it later.

Buy this course · ₹399See what is inside

Or open everything with a plan