IL-0478 Amazon · 2026-05-18 (published)ai-infranetworking
A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps
The Guard paper by Amazon's Store Foundational AI team describes straggler detection and node health management for large-scale training on clusters of thousands of GPUs in 8-GPU nodes, with NVLink inside nodes and external network cards between them. It calls machines that pass health checks for hard failures but run slower in real workloads grey nodes, and slow communication from a failed network adapter is one cause it describes. NCCL falls back to another GPU's adapter, usually GPU 0's, without a hardware alert; the job keeps running, but that adapter carries about double the traffic, each training step takes about 0.3 seconds longer, and resolving the degraded path brought step time from 8.7 s back to 8.4 s. In MoE models the delay comes back at every synchronization phase and adds up over training.
Guard finds such nodes in two steps. Online monitoring, collected through NVIDIA DCGM and sidecar processes every 30 seconds to 1 minute, tracks GPU and network metrics including network device status, error counts and transmission rate, compares each node with its peers in the same job and treats training step time as the main signal; a node that deviates on several indicators for consecutive windows is removed from the healthy pool. An offline node sweep then runs single-node and multi-node workloads for 1–2 hours before the node may return, and misrouted traffic shows at once as higher latency or lower bandwidth even in a 2-node sweep. Nodes that fail the sweep stay quarantined for repair or re-imaging.
- Severity
- –
- Scope
- Large-scale MoE foundation model training: each training step about 0.3 s slower (8.7 s instead of 8.4 s)
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
In Amazon's large-scale foundation model training, training steps took about 0.3 seconds longer (8.7 s instead of 8.4 s) when a node had a GPU whose network adapter had failed. No hardware failure alert fired, and standard node checks such as GPU burn-in and short NCCL tests rarely catch such nodes.
NCCL normally gives each GPU its own network adapter. When a GPU's adapter fails, NCCL sends that GPU's traffic through another GPU's adapter, typically GPU 0's; in the paper's Table 1, GPU7 expected NIC 7 but actually used NIC 0 (misrouted), with the status "Faulty (adapter down)". The remaining adapter then carried about double the network traffic, which lowered effective bandwidth and made routing asymmetric.
The slowdown was worst in Mixture of Experts (MoE) training, where token dispatch and result combination both communicate across the whole expert-parallel group, and in a model with 32 MoE layers the delay repeats in every layer. Because the rerouting keeps communication working, the node stays in the job and keeps lagging behind its peers.
Symptom Throughput degradation
Root cause
Root cause Hardware failure
- Trigger
- The network adapter of one GPU on a node failed.
Detection
No hardware alert or standard health check showed the fault. It showed as slower training steps, in the per-node network device status and transmission rate that Guard monitors, and as elevated latency or reduced bandwidth in a 2-node sweep.
Remediation
- Resolved the degraded communication path, which brought training step time from 8.7 s back to 8.4 s
Action items
- Monitor each node's network device status continuously to find and isolate nodes whose traffic NCCL rerouted
- Remove a node flagged by online monitoring from the healthy node pool and schedule it for offline verification
- Run a multi-node sweep that stresses cross-node collective communication to expose degraded links and misrouted traffic
- Trigger node sweeps after anomalies found by online monitoring and after repairs, before a node re-enters production
Lessons
NCCL's transparent rerouting keeps both the job and NCCL tests working while it hides a failed adapter, so checks that verify functional correctness do not find the node. Amazon relies on performance-aware monitoring during training, with step time as the main signal, and on offline verification under realistic load before nodes return to service.
Sources
References
Related cases
IL-0586Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minuteOpenAI · 2026same root cause · shared NIC, cable and switch port link
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared NCCL
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared NCCL
IL-0418Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputersGoogle · 2024same root cause · shared NIC, cable and switch port link
IL-0239LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024same root cause · shared NCCL
IL-0540Korea's local government administrative services and Government24 went down as a router port dropped most large packetsKorea Ministry of the Interior and Safety · 2023same root cause · shared NIC, cable and switch port link
IL-0125A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022same root cause · shared NCCL
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0478
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0478/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.