Incident: Weights that never moved, and no error

Module 5 · Incident file · About 40 minutes

What happened

Elana Simon published "the bug that taught me more about PyTorch than years of using it" on 22 October 2025 (elanapearl.github.io). She was training a small two-layer sparse autoencoder on an M3 MacBook Pro, on Apple's GPU backend, MPS. "My training loss plateaued and wouldn't budge." She tried "every hyperparameter combination, rewrote my loss function, spent days assuming I'd made some stupid mistake."

Then she looked at the weights. The decoder's were changing. "But the encoder weights… weren't updating at all. No NaNs, no suspicious patterns… they just… weren't changing." Gradients reached both layers. Inside Adam, the encoder's running average of squared gradients, exp_avg_sq, had a maximum of 0; the decoder's was 1.18e+11. On the CPU, in float32, the same code trained normally, so the fault belonged to the device, not the precision.

The cause was a PyTorch bug in the MPS kernels for addcmul_ and addcdiv_, which Adam uses to update its state and the weights: they "silently fail when writing to non-contiguous output tensors". Her code was on PyTorch 2.2.1; the bug "was already fixed in v2.4". Her fixes: make weights contiguous at initialisation, upgrade PyTorch, or upgrade to macOS 15, where MPS "handles non-contiguous tensors natively". The cost was days of a stalled research project, and nothing ever raised an error.