GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable

Symptoms

The GLM-130B pretraining run on 96 nodes, started on 2022-05-06 at 04:00, was interrupted again and again by single nodes that broke down, lost connection or reported missing GPUs, and each time training had to be restarted from a saved step. On 2022-05-30 a restart got stuck in data loading because of a Lustre file system error, and the 2.3T text corpora could no longer be read.

The training notes record each interruption. On 2022-05-07 at 20:14 nodes n30041 and n30157 broke down. On 2022-05-11 at 12:13 n30115 broke down, and on 2022-05-20 at 00:03 n30066 broke down. On 2022-05-22 at 19:27 n30126 lost connection, and on 2022-05-26 at 04:30 n30039 reported missing GPUs. On 2022-06-02 n30145 had a CPU error, on 2022-06-08 at 08:00 another node failed, and on 2022-06-09 at 13:30 the training terminated unexpectedly, which the team suspected was a network communication problem. On 2022-06-19 at 00:10 n30085 broke down, and on 2022-06-26 at 16:00 a node failed with "Unexpected NVLink Error".

On 2022-05-21 the training flop was only 127T, lower than before; the speed test before the run had measured 134TFLOPS at 88.5s/iter for a batch of 4224. The team suspected that the replacement node n30076 had an unknown fault and removed it from step 16600, but the speed did not change.

On 2022-05-30 at 14:00 the team restarted from global_step23200 and found the restart stuck in data loading. The cause was an error of the Lustre file system: the 2.3T text corpora could not be read, the engineer could not recover the data, and the data had to be copied from a backup disk to the file system again, which took a few days.

Symptom Training job failureNode NotReadyGPU unavailableStorage disconnected or stale mountsThroughput degradation

Root cause

Trigger

Individual nodes of the 96-node allocation failed during training, through breakdowns, a lost connection, missing GPUs, a CPU error and an NVLink error, and on 2022-05-30 the Lustre file system that held the training corpus failed.

Detection

The Lustre error was found when a restart from global_step23200 stayed stuck in data loading.

Remediation

• Restarted the training from the last saved step after each node failure
• Changed the checkpoint saving interval from 500 steps to 100 steps
• Switched the job to another node pool and restarted from step 15600
• Removed the suspected replacement node n30076 from step 16600, which did not change the speed
• Copied the 2.3T text corpora from a backup disk back to the Lustre file system, which took a few days

Action items

–

Lessons

–

Sources

• https://github.com/THUDM/GLM-130B/blob/main/logs/main-log-en.md · github · archived copy

Related cases

• IL-0416 Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026

  same root cause · shared GPU hardware and memory

• IL-0415 GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025

  same root cause · shared GPU hardware and memory

• IL-0482 Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025

  same root cause · shared GPU hardware and memory

• IL-0421 Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025

  same root cause · shared GPU hardware and memory

• IL-0414 Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025

  same root cause · shared GPU hardware and memory

• IL-0411 Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024

  same root cause · shared GPU hardware and memory

• IL-0238 Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024

  same root cause · shared GPU hardware and memory

• IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024

  same root cause · shared GPU hardware and memory

• IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024

  same root cause · shared GPU hardware and memory

• IL-0427 A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021

  same root cause · shared GPU hardware and memory

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0429
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0429/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.