IL-0239 Shanghai AI Laboratory · 2024-04-03 (published)ai-infra
LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errors
Acme's Seren (2,288 GPUs, scheduler built on Slurm) and Kalos (2,416 GPUs, scheduler built on Kubernetes) clusters have nodes with 8 NVIDIA A100-SXM 80GB GPUs linked by NVLink and NVSwitch and 200Gbps HDR InfiniBand between nodes, and they are used to develop LLMs from 7B to over 123B parameters with the InternEvo framework. The paper classifies job failures from runtime logs into infrastructure, framework and script failures. Script and framework errors are the most numerous but happen early and are fixed by changing code or configuration; infrastructure failures happen midway through long pretraining jobs, can hit one job several times, and need careful diagnosis that often ends in maintaining or replacing hardware, which makes restarts expensive. About 40% of all jobs fail, and jobs that complete use only 20-30% of GPU resources. Evaluation jobs rarely fail: only 6.7% of evaluation tasks in Kalos hit an error and none hit a GPU or NVLink failure. After the overheating seen in July 2023 the team improved the cluster's cooling, and such failures became much less frequent.
At first developers took turns on call and restarted failed pretraining jobs by hand, each restart going back to the last checkpoint; for the 104B model, reloading checkpoints cost much training progress, and for the 123B model a month later the team saved checkpoints more often and let jobs terminate gracefully, saving their state before they stopped. The team then built a fault-tolerant pretraining system: asynchronous checkpointing that copies model state to host memory and persists it from a separate thread, cutting checkpoint overhead by 3.6-58.7 times; failure diagnosis that compresses logs with regular-expression filter rules written by an LLM agent and then finds the root cause with predefined rules or, failing those, with GPT-4 over a vector store of logs; and recovery that, for NVLink errors, runs a two-round NCCL allgather test on pairs of nodes to find and cordon faulty nodes, and, for a loss spike, restarts from an earlier healthy checkpoint and skips the following data batches. The failure diagnosis system reduced manual intervention by around 90%. For the framework problems, the team fixed garbage collection to set intervals on each rank and avoided the dataloader leak by setting num_worker=0 and enabling garbage collection in the dataloader.
- Severity
- –
- Scope
- Pretraining jobs on Seren and Kalos failed or stalled and lost progress back to the last checkpoint; infrastructure failures took over 82% of the GPU time of failed jobs, and about 40% of all jobs failed
- Users affected
- LLM developers at Shanghai AI Laboratory
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
Over six months from March to August 2023, LLM pretraining jobs on the Seren and Kalos clusters of Shanghai AI Laboratory's Acme datacenter, 4,704 A100 GPUs in total, failed again and again partway through and had to be restarted from the last checkpoint. Jobs that failed on infrastructure problems were only 11% of failed jobs by count but used over 82% of the GPU time of failed jobs.
The failures showed in the runtime logs (stdout and stderr) of the LLM framework. Among infrastructure failures, NVLink Error occurred 54 times, with an average GPU demand of 800, an average time to failure of 868.1 minutes and 30.25% of failed GPU time, and took 95.6 minutes on average to restart; CUDA Error 21 times (15.77%); Node Failure, meaning errors from unclear hardware problems, 16 times (14.30%); ECC Error 12 times (11.00%); Network Error 12 times (4.53%) with an average restart time of 592.1 minutes; Connection Error 147 times (3.44%); S3 Storage Error 10 times (2.12%); NCCL Timeout Error 6 times (0.50%); and NCCL Remote Error 3 times (0.15%). Logs of pretraining jobs reach hundreds of MBs, and one failure can print several errors at once: a job could fail with NCCLTimeoutError, CUDAError and several kinds of RuntimeError when the root cause was the CUDAError.
Jobs also had to be restarted for loss spikes, a sudden rise in a training loss that had been falling and that does not recover for some time, and for training that got stuck: some jobs stalled on infrastructure problems without printing any error and were found only when users inspected them by hand. Training 7B models in Kalos tended to overheat GPUs and cause NVLinkError or ECCError; the server room temperature rose by about 5°C during these runs, most of which took place in July 2023, the hottest month on record. Many ConnectionError and NetworkError failures came from auxiliary services for metric reporting, logging, monitoring and alerting that the framework connects to.
Two framework problems also had to be troubleshot. In LLM jobs written in Python, irregular slowdowns cut the average throughput per step by 2-3 times in severe cases, and Linux perf showed the garbage collection function list_traverse taking 30% of each step. A Dataloader Worker Killed error appeared on average 27 hours after a task started, from a gradual memory leak in PyTorch's dataloader with num_worker>0.
Symptom Training job failureGPU unavailableNCCL timeoutTraining divergence or loss spike
Root cause
Root cause Hardware failure
- Trigger
- –
Detection
Failures were found in the framework's runtime logs, later by a diagnosis system that combines rules with an LLM; the pretraining framework detects loss spikes automatically. Jobs that stalled without an error were noticed only when users inspected them by hand.
Remediation
- Restarted interrupted pretraining jobs by hand from the last checkpoint, with developers on call in turn
- Saved checkpoints at shorter intervals and let jobs terminate gracefully to keep their current state
- Improved the cluster's cooling after GPU overheating caused NVLink and ECC errors
- Found faulty nodes with a two-round NCCL allgather test and cordoned them off
- Restarted from an earlier healthy checkpoint and skipped the following data batches after a loss spike
- Fixed the garbage collection interval on each rank in InternEvo V2
- Set num_worker=0 and enabled garbage collection in the dataloader to stop the memory leak
Action items
- Move failure diagnosis from GPT-4 to Shanghai AI Laboratory's own LLMs
- Upgrade the infrastructure, with a particular focus on the NICs, and expand the computing cluster for larger-scale pretraining
Lessons
Infrastructure failures are rare by count but cost the most GPU time, so diagnosing them and recovering quickly matters most for training efficiency. Matching error logs with regular-expression rules alone is often inaccurate because the logs are diverse and one failure can print several errors. Jobs that stall without an error or that are paused for loss spikes waste resources until someone looks, which calls for a failure-handling system that finds and fixes faulty jobs by itself.
Sources
References
Related cases
IL-0421Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025same root cause · shared NCCL, GPU hardware and memory
IL-0127Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024same root cause · shared NCCL, GPU hardware and memory
IL-0416Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026same root cause · shared GPU hardware and memory
IL-0478A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training stepsAmazon · 2026same root cause · shared NCCL
IL-0415GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025same root cause · shared GPU hardware and memory
IL-0482Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025same root cause · shared GPU hardware and memory
IL-0414Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025same root cause · shared GPU hardware and memory
IL-0411Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checksMeta · 2024same root cause · shared GPU hardware and memory
IL-0238Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024same root cause · shared GPU hardware and memory
IL-0381A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeoutsLinode · 2022same root cause · shared Object storage
IL-0525OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenanceOpenStreetMap Foundation · 2022same root cause · shared Data center power and cooling
IL-0125A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022same root cause · shared NCCL
IL-0429GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022same root cause · shared GPU hardware and memory
IL-0427A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021same root cause · shared GPU hardware and memory
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0239
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0239/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.