Incident Library

IL-0479 Amazon Web Services · 2025-02-17 (published)ai-infra

Accelerator nodes with silent data corruption changed LLM training outputs, gradients and weights with no error signal

The paper, by authors at Amazon Web Services and Harvard University, studies silent data corruption (SDC), hardware producing incorrect results without any failure signal, on 15 unhealthy nodes that a cloud computing platform's fleet management had taken out of production because they showed SDC, and compares them with healthy production nodes. After a node is installed in a data center, the fleet management flow runs full-node collective communication stress tests, compute unit stress tests and a small LLM training run whose outputs are compared with precomputed golden values, and it runs them again when a node comes back from customer use.

The authors trained identical models on each unhealthy and healthy pair with deterministic execution from the XLA compiler, so that every difference came from SDC. They compared submodule outputs in a lock-step setup that overwrote the unhealthy node's tensors with the healthy node's after each self-attention and FFN module, compared gradients per optimizer step by copying the healthy parameters after every step, and then let whole pre-training and fine-tuning runs diverge. Mismatches were usually rare and the gradient noise small, at worst an L2 norm of the gradient difference 5.1% of the true gradient's on Node 11, but the weights converged to different optima and fine-tuning on some nodes had loss spikes that in one case fully corrupted the model. Checksummed matrix multiplication (algorithm-based fault tolerance, ABFT) flagged errors frequently only on Node 14, and more synchronization lowered accelerator utilization, which made SDC less frequent. The unhealthy nodes were held only for the study and are being repaired and returned to the production pool.

Severity
–
Scope
15 unhealthy accelerator nodes studied: LLM training on them produced different weights, loss spikes in some fine-tuning runs and in one case a model with zero test accuracy on CosmosQA
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

Machine learning accelerator nodes that a cloud computing platform's automated fleet management had removed from production produced wrong computation results without any error signal. When the same LLM training ran on one of them and on a healthy node, the training loss was identical, but submodule outputs, gradients and model weights differed.

Some of the nodes had been healthy at first and began to fail the fleet's pre-checks after they had been used in training. The fleet's tests can only run while a node is not in customer use, so a node that degrades during a multi-month training run can corrupt training before fleet management isolates it.

The corruption differed from node to node and over time. In the submodule comparison, Nodes 10 and 11 had high mismatch frequency (2.89e-2 of elements in the forward attention computation on Node 11 and 4.78e-3 on Node 10), Nodes 2 and 3 showed none in that setting, and on Node 9 the mismatch severity exceeded 100, so some values differed by large factors. On Nodes 7 and 14 mismatches came in occasional spikes with none during most of the training time, and Nodes 10 and 11 had a high peak in the first steps of the run.

Over whole runs the effect accumulated. During pre-training the weights on the unhealthy nodes drifted away from the healthy node's while the loss curves stayed identical; on Node 13 there was no parameter difference until step 450, after which the weights quickly drifted. In fine-tuning Mistral-7B-v0.3 on CosmosQA, loss spikes appeared on some unhealthy nodes, and on Node 6 a spike near the end left the fine-tuned model with zero test accuracy on CosmosQA.

Symptom Training divergence or loss spike

Root cause

Root cause Hardware failure

Component –
Contributing factor Monitoring blind spot
Trigger
–

Detection

The fleet management tests found the nodes: collective communication and compute unit stress tests and a small LLM training run compared with golden values, run after installation and when a node returns from customer use. Training loss showed nothing on most nodes.

Remediation

Action items

–

Lessons

SDC can change training for an unknown period before any loss spike appears, so watching the training loss is not enough. ABFT did not detect it reliably; the authors' possible explanations are that mismatches were too rare and small to exceed floating-point error bounds, that ABFT lowered accelerator utilization and with it the SDC rate, and that SDC may arise outside matrix multiplication; the authors suggest recomputation, for example a shadow data-parallel replica whose gradients are compared with a target rank's during the gradient all-reduce, and point at reducing the sharpness of the loss surface to limit the damage.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.