Cases1 case

1 to 1 of 1

1. IL-0479 Accelerator nodes with silent data corruption changed LLM training outputs, gradients and weights with no error signal

   Amazon Web Services2025

   Symptoms

   Machine learning accelerator nodes that a cloud computing platform's automated fleet management had removed from production produced wrong computation results without any error signal. When the same LLM training ran on one of them and on a healthy node, the training loss was identical, but submodule outputs, gradients and model weights differed.

   Training divergence or loss spike

   Root cause

   Hardware failure

   Component

   –

   Contributing factor

   Monitoring blind spot

   Detection

   The fleet management tests found the nodes: collective communication and compute unit stress tests and a small LLM training run compared with golden values, run after installation and when a node returns from customer use. Training loss showed nothing on most nodes.

   Remediation

   • Kept the nodes that failed the fleet's SDC tests out of production availability
   • Sent the nodes held for the study for repair before returning them to the production pool

   Lessons

   SDC can change training for an unknown period before any loss spike appears, so watching the training loss is not enough. ABFT did not detect it reliably; the authors' possible explanations are that mismatches were too rare and small to exceed floating-point error bounds, that ABFT lowered accelerator utilization and with it the SDC rate, and that SDC may arise outside matrix multiplication; the authors suggest recomputation, for example a shadow data-parallel replica whose gradients are compared with a target rank's during the gradient all-reduce, and point at reducing the sharpness of the loss surface to limit the damage.

   View case

By root cause1