Incident Library

Organisation

DeepSeek

DeepSeek · deepseek

1 case · 2024

Root cause Hardware failure
Component GPU hardware and memoryInfiniBand fabric
Contributing factor –
Symptom GPU unavailableLink flappingTraining divergence or loss spikeTraining job failure

Cases

  1. IL-0238 Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024

    Root cause Hardware failure