Incident Library

Organisation

Tsinghua University

Tsinghua University · tsinghua-university

1 case · 2022

Root cause Hardware failure
Component GPU hardware and memoryShared network filesystem
Contributing factor –
Symptom GPU unavailableNode NotReadyStorage disconnected or stale mountsThroughput degradationTraining job failure

Cases

  1. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022

    Root cause Hardware failure