Incident: Faulty chips that trained models quietly wrong

Module 9 · Incident file · About 35 minutes

What happened

In February 2025 Jeffrey Ma, Hengzhi Pei, Leonard Lausen and George Karypis (Harvard and Amazon Web Services) published "Understanding Silent Data Corruption in LLM Training" (arxiv.org/abs/2502.12340, ACL 2025). Silent data corruption (SDC) is "hardware produces incorrect computations without explicit failure signals": no crash, no error, a wrong number.

Their material was real faulty hardware. Working "with a cloud computing platform", they used 15 nodes that had "failed production margin stress tests"; the team held them "only temporarily" before repair and compared each with healthy nodes, using deterministic execution so that any difference was the hardware's.

What they found is what makes SDC dangerous. On the worst node, the gradient error was "5.1% of that of the ground-truth gradients". In pretraining, losses stayed nearly identical to the healthy run while the weights drifted apart; one node showed no difference for 450 steps, then began to drift. In fine-tuning, three nodes produced loss spikes, and on one "the loss spike occurs near the end of fine-tuning, which leads to the resulting model having zero test accuracy". The paper's introduction cites the scale: six unplanned interruptions blamed on SDC in Llama 3's 54-day snapshot, and Google's estimate of an SDC event "every week or two" in Gemini training.