Incident Library

Organisation

ByteDance

ByteDance · bytedance

1 case · 2024

Root cause DNS or network fault
Component NCCLNIC, cable and switch port link
Contributing factor Unreviewed defaults
Symptom Link flappingNCCL timeoutThroughput degradation

Cases

  1. IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUsByteDance · 2024

    Root cause DNS or network fault