IL-0238 DeepSeek · 2024-08-31 (published)ai-infra
Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 cluster
Fire-Flyer 2 is DeepSeek's deep learning training cluster: about 1,250 GPU nodes, each with 8 NVIDIA PCIe A100 40GB GPUs and a single 200Gbps Mellanox ConnectX-6 InfiniBand NIC, and nearly 200 storage servers, on a two-zone two-layer fat-tree InfiniBand network that carries both computation and storage traffic. When LLM training arrived, DeepSeek added an NVLink Bridge between pairs of PCIe GPUs for tensor parallelism. The paper reports a year of hardware failure data from this cluster. Xid 74, the most common error at 42.57% of all Xid errors, occurs mainly on the NVLink Bridge; its high rate comes from the inherent fault rate of the NVLink Bridge connectors multiplied by thousands of GPUs, and apart from stress testing to weed out GPUs that keep repeating the error there is no good way to prevent it. The paper compares this with another study that found NVLink failures to be 52.42% (54 of 103) of failures.
For GPU memory ECC errors (Xid 63, 64, 94, 95), the row remapping of the A100 means most cases are cleared by resetting the GPU. Uncorrectable failures (Xid 44, 48, 61, 62, 69, 79) are reported to the user application and need a GPU reset or node reboot, and Xid 119 needs a fieldiag test and in most cases an RMA. Errors that ECC did not catch were the hardest: they produced gradnorm spikes, loss explosions and non-convergence. InfiniBand flash cuts broke communication and failed multi-node tasks at random times. To limit the cost of these failures, a checkpoint manager saves model parameters and optimizer state to the 3FS file system about every 5 minutes, so an interruption loses at most the last 5 minutes of training, and a validator runs weekly on every node, checking link speed and status, CPU and memory, each byte of GPU memory, GEMM at full memory, NVLink bandwidth through an intra-node allreduce and storage bandwidth, and removes faulty nodes from scheduling. Diagnosis tools such as hostping are integrated as well, but finding the root cause of hardware failures is still hard work for the operations team.
- Severity
- –
- Scope
- Training tasks on the Fire-Flyer 2 cluster were interrupted or failed, losing up to 5 minutes of progress per interruption; undetected memory errors caused gradnorm spikes, loss explosions and non-convergence; cluster utilization fell
- Users affected
- DeepSeek-AI model training
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
On DeepSeek's Fire-Flyer 2 training cluster of 10,000 PCIe A100 GPUs, Xid 74 NVLink errors, which occur mainly on the NVLink Bridge between pairs of GPUs, were the most common GPU Xid error and occurred several orders of magnitude more often than other hardware faults. InfiniBand link flash cuts disrupted application communication and sometimes failed training tasks, and computational and GPU memory errors that ECC did not detect caused gradnorm spikes, loss explosions and even non-convergence of models.
The Xid errors come from the NVIDIA driver and are logged in the kernel or event log of the operating system. Over one year the cluster logged 12970 of them: xid_74 5521 (42.57%), xid_43 4342 (33.48%), xid_31 2487 (19.18%), xid_45 240 (1.85%) and xid_13 45 (0.35%); the GPU memory ECC errors xid_63 245 (1.89%), xid_64 2, xid_94 13 and xid_95 17; the uncorrectable failures xid_44 1, xid_48 2, xid_61 13, xid_62 3, xid_69 1 and xid_79 37; and one xid_119, a failed GPU GSP module. Xid 13, 31, 43 and 45 point to illegal memory access or illegal instructions in user code, Xid 43 being illegal memory access, but they can also come from corrupted memory data.
InfiniBand link failures made up 30% of hardware faults when Xid 74 is left out. From October 2023 to March 2024 the cluster recorded 89 network flash cuts, 54 CPU memory ECC errors and 120 xid_63, 15 xid_79, 7 xid_94, 6 xid_95 and 1 xid_64 GPU errors, 292 in total; GPU ECC faults clearly outnumbered CPU ones. The daily record of InfiniBand flash cuts from April 2023 to March 2024 shows them scattered randomly across the whole period, with up to 10 on a single day (2023/5/28, 2023/7/7 and 2023/7/12). Because most tasks span many nodes, a flash cut on one node affected many others and lowered cluster utilization.
Symptom GPU unavailableLink flappingTraining job failureTraining divergence or loss spike
Root cause
Root cause Hardware failure
- Trigger
- –
Detection
Xid errors were read from the NVIDIA driver's messages in the kernel or event log. A validator run weekly by the automatic operations system tested every node, and diagnosis tools such as hostping were integrated into the platform.
Remediation
- Reset the GPU to clear most GPU memory ECC errors, relying on A100 row remapping
- Reset the GPU or rebooted the node after uncorrectable GPU errors
- Ran a fieldiag test after Xid 119 and sent most such GPUs for RMA
- Removed nodes that failed the weekly validator from the scheduling platform
- Resumed interrupted training from checkpoints saved about every 5 minutes
Action items
–
Lessons
The best way to keep devices stable is to find problems before they happen, which is why the validator runs weekly. GPU ECC faults far outnumber CPU ones and need to be handled quickly so that performance and accuracy are not affected. Software-looking Xid errors such as illegal memory access may come from corrupted memory, so hardware faults should be considered once software bugs are ruled out. Whether to install NVLink should depend on actual need because of its drawbacks.
Sources
References
Related cases
IL-0416Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026same root cause · shared GPU hardware and memory
IL-0415GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025same root cause · shared GPU hardware and memory
IL-0482Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025same root cause · shared GPU hardware and memory
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared GPU hardware and memory
IL-0476PCIe downgrade on one machine slowed a 128-machine V100 training job for 40 minutes through a PFC surgeByteDance · 2025same root cause · shared InfiniBand fabric
IL-0414Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025same root cause · shared GPU hardware and memory
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared GPU hardware and memory
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared GPU hardware and memory
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
IL-0427A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0238
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0238/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.