Cases1 case

1 to 1 of 1

1. IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errors

   Shanghai AI Laboratory2024

   Symptoms

   Over six months from March to August 2023, LLM pretraining jobs on the Seren and Kalos clusters of Shanghai AI Laboratory's Acme datacenter, 4,704 A100 GPUs in total, failed again and again partway through and had to be restarted from the last checkpoint. Jobs that failed on infrastructure problems were only 11% of failed jobs by count but used over 82% of the GPU time of failed jobs.

   Training job failureGPU unavailableNCCL timeoutTraining divergence or loss spike

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLTraining data loaderObject storageData center power and cooling

   Contributing factor

   –

   Detection

   Failures were found in the framework's runtime logs, later by a diagnosis system that combines rules with an LLM; the pretraining framework detects loss spikes automatically. Jobs that stalled without an error were noticed only when users inspected them by hand.

   Remediation

   • Restarted interrupted pretraining jobs by hand from the last checkpoint, with developers on call in turn
   • Saved checkpoints at shorter intervals and let jobs terminate gracefully to keep their current state
   • Improved the cluster's cooling after GPU overheating caused NVLink and ECC errors
   • Found faulty nodes with a two-round NCCL allgather test and cordoned them off
   • Restarted from an earlier healthy checkpoint and skipped the following data batches after a loss spike
   • Fixed the garbage collection interval on each rank in InternEvo V2
   • Set num_worker=0 and enabled garbage collection in the dataloader to stop the memory leak

   Action items

   • Move failure diagnosis from GPT-4 to Shanghai AI Laboratory's own LLMs
   • Upgrade the infrastructure, with a particular focus on the NICs, and expand the computing cluster for larger-scale pretraining

   Lessons

   Infrastructure failures are rare by count but cost the most GPU time, so diagnosing them and recovering quickly matters most for training efficiency. Matching error logs with regular-expression rules alone is often inaccurate because the logs are diverse and one failure can print several errors. Jobs that stall without an error or that are paused for loss spikes waste resources until someone looks, which calls for a failure-handling system that finds and fixes faulty jobs by itself.

   View case

By root cause1