Cases1 case

1 to 1 of 1

1. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count

   Hugging Face2025

   Symptoms

   Within hours of launching the full SmolLM3 pretraining run (3B parameters, 11T tokens, 384 H100 GPUs on 48 nodes), training throughput fell steeply, with repeated sharp drops. The team expected a steady 13,500 to 14,000 tokens per second, and none of the throughput ablations run beforehand on the same 48 nodes had shown drops.

   Throughput degradation

   Trigger

   Launching the full run with the whole ~24 TB dataset on shared WekaFS storage and the real step count for 11T tokens, both far larger than in the ablation runs.

   Root cause

   Capacity exhaustion

   Component

   Shared network filesystemTraining data loader

   Contributing factor

   Overlapping failuresUnrepresentative stagingUnverified assumption

   Detection

   The team watched training throughput in tokens per second and saw it fall within hours of launch; node-monitoring metrics then tied the large fall to spikes in disk read latency.

   Remediation

   • moved the training data from WekaFS to the local NVMe RAID scratch storage on each node
   • copied the dataset to replacement nodes from a healthy node with fpsync, cutting the copy from 3 hours to 1.5 hours
   • kept a spare node with the dataset preloaded in the Slurm reservation to swap in when a node failed
   • reproduced the drops on a single node and with step counts of about 3M and 32k to rule out hardware and isolate the step count
   • replaced nanotron's nanosets dataloader with the TokenizedBytes dataloader copied from the SmolLM2 setup, which removed the drops
   • pre-shuffled the tokenized sequences offline for each epoch with different seeds (about 1 hour) to fix the noisier loss from TokenizedBytes

   Lessons

   Ablations on data subsets and a short step horizon did not exercise either cause: a shared cache-and-evict filesystem near its capacity, and a dataloader index that grows with the step count. Reproducing the drops on one node separated hardware from software, and varying only the step count pointed at the dataloader. The team chose proven replacements over repairs under deadline pressure, taking an existing dataloader instead of fixing nanosets and offline shuffling instead of changing the dataloader, and warns that too many such shortcuts leave a patchwork system that is hard to maintain.

   View case

By root cause1