Incident Library

Organisation

Alibaba Group

Alibaba Group · alibaba-group

2 cases · 2024 to 2025

Root cause Capacity exhaustion1Hardware failure1
Component GPU hardware and memory2InfiniBand fabric1NCCL1PyTorch1
Contributing factor Monitoring blind spot2Unverified assumption1Overlapping failures1
Symptom GPU unavailable1NCCL timeout1Throughput degradation1Training job failure1

Cases

  1. IL-0421 Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025

    Root cause Hardware failure

  2. IL-0249 Large training jobs on a cluster of over 10,000 GPUs slowed for tens of seconds to nearly ten hours without crashingAlibaba Group · 2024

    Root cause Capacity exhaustion