Cases2 cases

1 to 2 of 2

1. IL-0586 Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minute

   OpenAI2026

   Symptoms

   During a 50K GPU production pretraining job at OpenAI on Cluster A, all four links of one optical transceiver on a T0 switch flapped in rapid succession, and the job's throughput dropped by about 25% for the minute of flaps. The four links went to NICs on four different nodes, three of which were running in the job.

   Link flappingThroughput degradation

   Trigger

   An optical transceiver on a T0 switch had a glitch.

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   –

   Detection

   The NICs detected their links going down and the T0 switch recorded the same links as down; the event is shown in the job's throughput and per-node packet loss graphs.

   Remediation

   • MRC recovered the packets lost during the flaps quickly, so the impact on the job stayed relatively small

   Lessons

   With MRC, failures of T0-to-T1 links and T1 switches have almost no effect on training, but NIC-to-T0 links and T0 switches still have a measurable effect and need more care. Most NIC-to-T0 link losses are transient, and a job can often recover to full speed quickly without evicting nodes.

   View case
2. IL-0238 Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 cluster

   DeepSeek2024

   Symptoms

   On DeepSeek's Fire-Flyer 2 training cluster of 10,000 PCIe A100 GPUs, Xid 74 NVLink errors, which occur mainly on the NVLink Bridge between pairs of GPUs, were the most common GPU Xid error and occurred several orders of magnitude more often than other hardware faults. InfiniBand link flash cuts disrupted application communication and sometimes failed training tasks, and computational and GPU memory errors that ECC did not detect caused gradnorm spikes, loss explosions and even non-convergence of models.

   GPU unavailableLink flappingTraining job failureTraining divergence or loss spike

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryInfiniBand fabric

   Contributing factor

   –

   Detection

   Xid errors were read from the NVIDIA driver's messages in the kernel or event log. A validator run weekly by the automatic operations system tested every node, and diagnosis tools such as hostping were integrated into the platform.

   Remediation

   • Reset the GPU to clear most GPU memory ECC errors, relying on A100 row remapping
   • Reset the GPU or rebooted the node after uncorrectable GPU errors
   • Ran a fieldiag test after Xid 119 and sent most such GPUs for RMA
   • Removed nodes that failed the weekly validator from the scheduling platform
   • Resumed interrupted training from checkpoints saved about every 5 minutes

   Lessons

   The best way to keep devices stable is to find problems before they happen, which is why the validator runs weekly. GPU ECC faults far outnumber CPU ones and need to be handled quickly so that performance and accuracy are not affected. Software-looking Xid errors such as illegal memory access may come from corrupted memory, so hardware faults should be considered once software bugs are ruled out. Whether to install NVLink should depend on actual need because of its drawbacks.

   View case

By root cause1