Cases8 cases

1 to 5 of 8

1. IL-0477 Slow compute on 8 ranks pushed a 10,000-GPU MoE job from 30 s to over 90 s per step; an alert pointed at the network

   Tencent2026

   Symptoms

   In an MoE training job of about 10,000 GPUs on Tencent's production cluster, iteration time rose from about 30 s to over 90 s and the ARGUS iteration-level detector raised an alert. At the same time the out-of-band monitoring system reported a "server port down" event on one of the affected nodes.

   Throughput degradation

   Root cause

   Hardware failure

   Component

   –

   Contributing factor

   Unverified assumption

   Detection

   ARGUS level L1 detected the iteration-time regression and raised an alert, and level L2's cross-rank comparison of phase durations located the 8 slow ranks.

   Remediation

   • Replaced the affected nodes, after which training speed recovered immediately

   Lessons

   Out-of-band infrastructure metrics can point at the wrong component: a communication anomaly or port event may be a secondary effect of compute degradation that propagates through collectives. Fine-grained tracing that separates compute operators from communication is what showed the network explanation was wrong.

   View case
2. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps

   Amazon2026

   Symptoms

   In Amazon's large-scale foundation model training, training steps took about 0.3 seconds longer (8.7 s instead of 8.4 s) when a node had a GPU whose network adapter had failed. No hardware failure alert fired, and standard node checks such as GPU burn-in and short NCCL tests rarely catch such nodes.

   Throughput degradation

   Trigger

   The network adapter of one GPU on a node failed.

   Root cause

   Hardware failure

   Component

   NCCLNIC, cable and switch port link

   Contributing factor

   Monitoring blind spot

   Detection

   No hardware alert or standard health check showed the fault. It showed as slower training steps, in the per-node network device status and transmission rate that Guard monitors, and as elevated latency or reduced bandwidth in a 2-node sweep.

   Remediation

   • Resolved the degraded communication path, which brought training step time from 8.7 s back to 8.4 s

   Action items

   • Monitor each node's network device status continuously to find and isolate nodes whose traffic NCCL rerouted
   • Remove a node flagged by online monitoring from the healthy node pool and schedule it for offline verification
   • Run a multi-node sweep that stresses cross-node collective communication to expose degraded links and misrouted traffic
   • Trigger node sweeps after anomalies found by online monitoring and after repairs, before a node re-enters production

   Lessons

   NCCL's transparent rerouting keeps both the job and NCCL tests working while it hides a failed adapter, so checks that verify functional correctness do not find the node. Amazon relies on performance-aware monitoring during training, with step time as the main signal, and on offline verification under realistic load before nodes return to service.

   View case
3. IL-0586 Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minute

   OpenAI2026

   Symptoms

   During a 50K GPU production pretraining job at OpenAI on Cluster A, all four links of one optical transceiver on a T0 switch flapped in rapid succession, and the job's throughput dropped by about 25% for the minute of flaps. The four links went to NICs on four different nodes, three of which were running in the job.

   Link flappingThroughput degradation

   Trigger

   An optical transceiver on a T0 switch had a glitch.

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   –

   Detection

   The NICs detected their links going down and the T0 switch recorded the same links as down; the event is shown in the job's throughput and per-node packet loss graphs.

   Remediation

   • MRC recovered the packets lost during the flaps quickly, so the impact on the job stayed relatively small

   Lessons

   With MRC, failures of T0-to-T1 links and T1 switches have almost no effect on training, but NIC-to-T0 links and T0 switches still have a measurable effect and need more care. Most NIC-to-T0 link losses are transient, and a job can often recover to full speed quickly without evicting nodes.

   View case
4. IL-0476 PCIe downgrade on one machine slowed a 128-machine V100 training job for 40 minutes through a PFC surge

   ByteDance2025

   Symptoms

   A distributed training task on 128 machines in ByteDance's production environment slowed down severely for 40 minutes but kept running, holding thousands of V100 GPUs at reduced speed. The alerting in place notified engineers only when a task stopped completely, so the slowdown itself raised no alert.

   Throughput degradation

   Trigger

   The PCIe link of one machine degraded from 6.4Gbps to 4Gbps during the training task.

   Root cause

   Hardware failure

   Component

   PCIe linkInfiniBand fabric

   Contributing factor

   No alert on limit

   Detection

   No alert fired because alerts came only when a task stopped. Engineers from several teams found the faulty machine by manual diagnosis, which took 40 minutes.

   Action items

   • Detect faulty machines automatically during training by comparing each machine's per-second monitoring metrics with the other machines (Minder)

   Lessons

   Some faults slow a task without stopping it, and alerting that waits for a halted task misses them. Logs alone are incomplete because they do not contain monitoring metrics such as PFC packet rates, and no single metric reveals every fault type, so ByteDance compares machines with each other on several metrics and treats a difference that lasts for minutes as a fault.

   View case
5. IL-0419 A Granite-20B training job on IBM's Vela ran almost 3x slower because one node's power card fault throttled its GPUs

   IBM2025

   Symptoms

   A Granite-20B training job running on 768 GPUs on IBM's Vela AI cluster suddenly showed a step time almost 3 times longer than before.

   Throughput degradation

   Trigger

   A power card issue in one of the job's 96 nodes made the node throttle its GPUs from about 400W to roughly 150W.

   Root cause

   Hardware failure

   Component

   Server power supply and power capping

   Contributing factor

   –

   Detection

   The team noticed the sudden rise in step time and debugged all 96 nodes of the job to find the node with the power card issue.

   Remediation

   • Debugged all 96 nodes of the job and found the node with the power card issue

   Action items

   • Add alerts at the infrastructure layer based on host VM logs, application logs and host BMC logs to find problem nodes quickly
   • Flag nodes with an active power break slowdown from nvidia-smi GPU counters in Autopilot and the Grafana dashboard

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed and the job restarts on healthy nodes. Alerts and health checks that locate such nodes let site reliability engineers tell users about infrastructure problems before the users notice them, and cut the time to find the root cause by over 2x.

   View case

By root cause1