Incident Library
Applied filters ComponentNIC, cable and switch port linkFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps
    Amazon2026
    Symptoms

    In Amazon's large-scale foundation model training, training steps took about 0.3 seconds longer (8.7 s instead of 8.4 s) when a node had a GPU whose network adapter had failed. No hardware failure alert fired, and standard node checks such as GPU burn-in and short NCCL tests rarely catch such nodes.

    Throughput degradation
    Trigger

    The network adapter of one GPU on a node failed.

    Root cause
    Hardware failure
    Component
    NCCLNIC, cable and switch port link
    Contributing factor
    Monitoring blind spot
    Detection

    No hardware alert or standard health check showed the fault. It showed as slower training steps, in the per-node network device status and transmission rate that Guard monitors, and as elevated latency or reduced bandwidth in a 2-node sweep.

    Remediation
    • Resolved the degraded communication path, which brought training step time from 8.7 s back to 8.4 s
    Action items
    • Monitor each node's network device status continuously to find and isolate nodes whose traffic NCCL rerouted
    • Remove a node flagged by online monitoring from the healthy node pool and schedule it for offline verification
    • Run a multi-node sweep that stresses cross-node collective communication to expose degraded links and misrouted traffic
    • Trigger node sweeps after anomalies found by online monitoring and after repairs, before a node re-enters production
    Lessons

    NCCL's transparent rerouting keeps both the job and NCCL tests working while it hides a failed adapter, so checks that verify functional correctness do not find the node. Amazon relies on performance-aware monitoring during training, with step time as the main signal, and on offline verification under realistic load before nodes return to service.

    View case
  2. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets
    Korea Ministry of the Interior and Safety2023
    Symptoms

    On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

    Service outagePacket lossHigh latency
    Root cause
    Hardware failure
    Component
    NIC, cable and switch port link
    Contributing factor
    Monitoring blind spotGray failure
    Remediation
    • Replaced the two malfunctioning L4 devices with higher-performance devices, tested them and resumed Government24 at 09:00 on 18 November
    • Moved the connection on the defective port of the Daejeon center router to another port at 07:00 on 19 November
    • Swapped out the module of the router that had the fault
    Action items
    • Inspect all old equipment that may have similar port defects
    • Revise the incident handling manual so the public is told about an outage quickly
    • Announce the status of service recovery quickly through various means
    • Build integrated monitoring of critical service systems and related equipment to catch signs of failure early
    • Form a standing incident response team to send into major outages
    • Prepare administrative measures and a response manual so administrative services can still be provided when core digital government services stop
    • Name a paralysis of the national computer network as a type of disaster and accident, managed from prevention to recovery
    • Build a government-wide crisis response system for linked information systems and strengthen the control tower for outage prevention and response
    • Improve how public IT projects are run and managed, replace hardware and software past their service life quickly and make project pricing realistic
    • Reduce the reliance on outsourcing for developing and operating information systems and bring digital talent into government
    • Review the redundancy, recovery systems and network configuration of the systems in the National Information Resources Service, and its organisation
    • Work with manufacturers on a way to find and monitor failures of internal parts earlier
    Lessons

    The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

    View case
Check your cluster for this

These 2 cases involve NIC, cable and switch port link and NCCL. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure2 cases

Component
NIC, cable and switch port link2NCCL1
Contributing factor
Monitoring blind spot2Gray failure1
Organisation
Amazon1Korea Ministry of the Interior and Safety1
  1. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps
    Amazon2026
  2. IL-0540 Korea's local government administrative services and Government24 went down as a router port dropped most large packets
    Korea Ministry of the Interior and Safety2023