Incident Library

Organisation

Amazon

Amazon · amazon

1 case · 2026

Root cause Hardware failure
Component NCCLNIC, cable and switch port link
Contributing factor Monitoring blind spot
Symptom Throughput degradation

Cases

  1. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training stepsAmazon · 2026

    Root cause Hardware failure