IL-0428 BigScience · 2022-03-04 (published)ai-infra

Test runs of 100B-200B models on 320 to 384 A100 GPUs hung once inter-node traffic grew, and each hang took nodes down

Before starting the tr11 176B training, BigScience ran test runs of 100B to 200B models with Megatron-LM and DeepSpeed on 320 to 384 A100 GPUs of the Jean Zay cluster. Above a certain scale the runs hung: random GPUs stopped responding, the job waited until it timed out, and each attempt left several nodes down. By growing the model first in layers and then in hidden size, the engineer showed that the hang appeared when inter-node traffic passed a certain level, and noted that DeepSpeed ZeRO shards data over all GPUs, so more GPUs means more communication. The hang lived partly in DeepSpeed and partly in Megatron-LM, where some processes stayed in a broadcast inside train_step while the others had moved on, but the cause could be FusedAdam, PyTorch, NCCL or the system itself.

Diagnosis was slow because every run of the hanging setup cost a few nodes, and with the engineer 10 hours behind Jean Zay nobody was there to reboot them. py-spy run through a second srun with --gres=gpu:0 could not reach several of the hung processes, so the engineer turned on Python's trace module in pretrain_gpt.py with a Tee helper that copies each process's output to its own file; this did not hide the hang and logged 14GB of Python calls from 320 processes.

Setting CUDA_LAUNCH_BLOCKING=1 made the hang go away, and in performance checks it had surprisingly little effect on throughput at this scale, although it normally slows training by making CUDA operations synchronous. The engineer paused the investigation to test the final version of the code without risking more nodes, and the team kept CUDA_LAUNCH_BLOCKING=1 as the workaround.

Severity: –
Scope: Test runs of 100B to 200B models on 320 to 384 A100 GPUs hung; each hanging run left nodes down until rebooted, with 13 of 52 nodes down at one point
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

On Jean Zay, test runs of large models across 320 to 384 A100 GPUs hung: random GPUs stopped responding and the whole job hung until it timed out. Every run of the hanging setup left a few nodes down until someone rebooted them, and at one point 13 of 52 nodes were down.

The hang depended on scale. A 200B model worked on 10 nodes and a model under 100B worked on 40 nodes (TP=8, PP=10, DP=4), but 100B over 20 to 40 nodes hung, reliably on 40 nodes and intermittently on 20. Keeping the hidden size at 1024*14 and adding layers, 10 layers (25B) and 20 layers (50B) worked and 40 layers (100B) hung after iteration 1 succeeded. Keeping 80 layers and growing the hidden size, 1024*10 (100B) worked and 1024*12 (145B) hung. The engineer read this as the system failing once inter-node traffic passed a certain level, not as a problem with the size of each all_reduce or broadcast.

On nodes 0 and 1, each running only tensor parallelism, py-spy showed 6 processes in "train (megatron/training.py:915)" and 2 processes in "broadcast (torch/distributed/distributed_c10d.py:1191)" called from "_aggregate_total_loss (deepspeed/runtime/pipe/engine.py:540)". Six processes had finished train_step while two never did, so the processes had fallen out of sync. When py-spy was run on all processes, several did not respond at all, so it could not show where those were hanging.

With asynchronous CUDA launches (CUDA_LAUNCH_BLOCKING=0), a 200B run on 160 GPUs with GBS 2048 "may hang" and on 320 GPUs "always hangs", while with CUDA_LAUNCH_BLOCKING=1 the 320-GPU run worked at 87.78 TFLOPs. Suspects included apex's FusedAdam, which had caused a serious issue a week earlier, PyTorch, NCCL and an internal problem of the system.

Symptom Training job failure NCCL timeout GPU unavailable Node NotReady

Root cause

Root cause unknown

Component NCCL PyTorch

Contributing factor Limited debug access

Trigger: Growing the test model past about 100B parameters over 20 to 40 nodes, which raised inter-node communication past the level the system handled, with CUDA launches left asynchronous.

Detection

–

Remediation

Set CUDA_LAUNCH_BLOCKING=1, which removed the hang with little effect on throughput at this scale

Action items

–

Lessons

Python's trace module is very useful for debugging hanging processes because it logs the last call each process made before it stopped responding, but it should be started later, just before train_step, to avoid gigabytes of traces from the dataloader and setup code.

Sources

https://github.com/bigscience-workshop/bigscience/blob/master/train/tr11-176B-ml/chronicles-prequel.md · github · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, NCCL, PyTorch among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0428
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0428/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.