IL-0417 ByteDance · 2025-05-12 (published)ai-infra

Stragglers slowed 42.5% of large LLM training jobs on ByteDance's cluster, mostly without any faulty server

The paper by ByteDance and NYU engineers studies traces of LLM pretraining jobs submitted from 2024/01/01 to 2024/05/31 on a dedicated training cluster with DGX-like servers (eight GPUs per server over NVLink or PCIe, four or eight NICs) on an over-provisioned three-layer CLOS network, where each job has exclusive GPUs. 3079 jobs that used at least 128 GPUs were analysed, 3.6% of them on at least 5000 GPUs, all running customised versions of Megatron-LM with data, pipeline, tensor and context parallelism. The in-house profiler NDTimeline recorded start and end times of compute and communication operations in 10% of training steps, and a what-if simulator replayed each job with straggling operations set to their ideal durations to estimate how long the job would take without stragglers; after dropping jobs restarted more than 15 times, jobs whose traces could not be analysed and traces with simulation error above 5%, the analysis covered 38.2% of the recorded jobs and 56.4% of their GPU hours.

Three causes explained most stragglers. First, the last pipeline stage also runs the loss layer, which takes more compute than a transformer layer, so splitting layers evenly over stages made that stage slower; for 39.3% of jobs the last stage caused most of the slowdown, and in a test job its forward and backward computation were 2.07 and 1.41 times slower than an average stage. Second, in long-context jobs the training data had a long-tailed sequence length distribution and microbatches were filled with random sequences up to the maximum length, but self-attention cost grows with the square of each sequence length, so some data-parallel ranks finished late each step; 21.4% of straggling jobs were affected, with an average slowdown of 1.34, and the effect grew with the maximum sequence length. Third, Python's stop-the-world garbage collector paused forward computation for hundreds of milliseconds at different times on different workers, and the pauses grew as jobs ran, which the team suspects comes from a memory leak. Two rarer causes were CUDA memory fragmentation that slowed PyTorch's allocator with many cudaFree and cudaMalloc calls, and reduce-scatter kernels blocking unrelated kernels in large MoE jobs, which the team suspects is a false dependency from unrelated kernels sharing the same CUDA hardware queue.

The team already assigns fewer layers to the last pipeline stage, but the number has to be tuned by hand and whole layers limit how evenly work can be split; manual tuning of one job gave a 9.9% speedup and still left the last stage's forward computation at 1.55 times that of the other stages. A planned garbage collection introduced in 2023 turns off automatic collection and runs it on all workers at a set step interval, which gave a 12.6% improvement on a job with 128 data-parallel ranks, but it is off by default because a wrong interval can run the job out of memory, so it is not widely used. A prototype that redistributes sequences between data-parallel ranks to balance computation improved throughput by 23.9% on a 32K job. The analysis was built into SMon, which alerts the on-call team and shows a heatmap of worker slowdowns whose pattern points to the cause; within a month it located faulty machines in three cases and found one sequence length case and one pipeline partitioning case.

Severity

–

Scope

42.5% of traced LLM pretraining jobs straggled; 10.4% of allocated GPU hours were wasted, and about 1% of jobs wasted at least 45.0% of their GPUs

Users affected

Teams that share ByteDance's internal LLM training cluster

Region

–

Time to detect

–

Time to resolve

–