Cases2 cases

1 to 2 of 2

1. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps

   Amazon2026

   Symptoms

   In Amazon's large-scale foundation model training, training steps took about 0.3 seconds longer (8.7 s instead of 8.4 s) when a node had a GPU whose network adapter had failed. No hardware failure alert fired, and standard node checks such as GPU burn-in and short NCCL tests rarely catch such nodes.

   Throughput degradation

   Trigger

   The network adapter of one GPU on a node failed.

   Root cause

   Hardware failure

   Component

   NCCLNIC, cable and switch port link

   Contributing factor

   Monitoring blind spot

   Detection

   No hardware alert or standard health check showed the fault. It showed as slower training steps, in the per-node network device status and transmission rate that Guard monitors, and as elevated latency or reduced bandwidth in a 2-node sweep.

   Remediation

   • Resolved the degraded communication path, which brought training step time from 8.7 s back to 8.4 s

   Action items

   • Monitor each node's network device status continuously to find and isolate nodes whose traffic NCCL rerouted
   • Remove a node flagged by online monitoring from the healthy node pool and schedule it for offline verification
   • Run a multi-node sweep that stresses cross-node collective communication to expose degraded links and misrouted traffic
   • Trigger node sweeps after anomalies found by online monitoring and after repairs, before a node re-enters production

   Lessons

   NCCL's transparent rerouting keeps both the job and NCCL tests working while it hides a failed adapter, so checks that verify functional correctness do not find the node. Amazon relies on performance-aware monitoring during training, with step time as the main signal, and on offline verification under realistic load before nodes return to service.

   View case
2. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case

By root cause1