Incident Library
Applied filters ComponentGPU hardware and memorySymptomNode NotReadyClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable
    Tsinghua University2022
    Symptoms

    The GLM-130B pretraining run on 96 nodes, started on 2022-05-06 at 04:00, was interrupted again and again by single nodes that broke down, lost connection or reported missing GPUs, and each time training had to be restarted from a saved step. On 2022-05-30 a restart got stuck in data loading because of a Lustre file system error, and the 2.3T text corpora could no longer be read.

    Training job failureNode NotReadyGPU unavailableStorage disconnected or stale mountsThroughput degradation
    Trigger

    Individual nodes of the 96-node allocation failed during training, through breakdowns, a lost connection, missing GPUs, a CPU error and an NVLink error, and on 2022-05-30 the Lustre file system that held the training corpus failed.

    Root cause
    Hardware failure
    Component
    GPU hardware and memoryShared network filesystem
    Contributing factor
    –
    Detection

    The Lustre error was found when a restart from global_step23200 stayed stuck in data loading.

    Remediation
    • Restarted the training from the last saved step after each node failure
    • Changed the checkpoint saving interval from 500 steps to 100 steps
    • Switched the job to another node pool and restarted from step 15600
    • Removed the suspected replacement node n30076 from step 16600, which did not change the speed
    • Copied the 2.3T text corpora from a backup disk back to the Lustre file system, which took a few days
    View case
Check your cluster for this

This case involves GPU hardware and memory and Shared network filesystem. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure1 case

Component
GPU hardware and memoryShared network filesystem
Contributing factor
–
Organisation
Tsinghua University
  1. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable
    Tsinghua University2022