Cases2 cases

1 to 2 of 2

1. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps

   Amazon2026

   Symptoms

   In Amazon's large-scale foundation model training, training steps took about 0.3 seconds longer (8.7 s instead of 8.4 s) when a node had a GPU whose network adapter had failed. No hardware failure alert fired, and standard node checks such as GPU burn-in and short NCCL tests rarely catch such nodes.

   Throughput degradation

   Trigger

   The network adapter of one GPU on a node failed.

   Root cause

   Hardware failure

   Component

   NCCLNIC, cable and switch port link

   Contributing factor

   Monitoring blind spot

   Detection

   No hardware alert or standard health check showed the fault. It showed as slower training steps, in the per-node network device status and transmission rate that Guard monitors, and as elevated latency or reduced bandwidth in a 2-node sweep.

   Remediation

   • Resolved the degraded communication path, which brought training step time from 8.7 s back to 8.4 s

   Action items

   • Monitor each node's network device status continuously to find and isolate nodes whose traffic NCCL rerouted
   • Remove a node flagged by online monitoring from the healthy node pool and schedule it for offline verification
   • Run a multi-node sweep that stresses cross-node collective communication to expose degraded links and misrouted traffic
   • Trigger node sweeps after anomalies found by online monitoring and after repairs, before a node re-enters production

   Lessons

   NCCL's transparent rerouting keeps both the job and NCCL tests working while it hides a failed adapter, so checks that verify functional correctness do not find the node. Amazon relies on performance-aware monitoring during training, with step time as the main signal, and on offline verification under realistic load before nodes return to service.

   View case
2. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets

   Korea Ministry of the Interior and Safety2023

   Symptoms

   On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

   Service outagePacket lossHigh latency

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   Monitoring blind spotGray failure

   Remediation

   • Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
   • Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
   • Swapped out the module of the router that had the fault

   Action items

   • Inspect all old equipment that may have similar port defects
   • Revise the incident handling manual so the public is told about an outage quickly
   • Announce the status of service recovery quickly through various means
   • Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
   • Form a standing incident response team to send into major outages
   • Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
   • Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
   • Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
   • Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
   • Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
   • Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
   • Work with manufacturers on a way to find and monitor failures of internal parts earlier

   Lessons

   The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

   View case

By root cause1