Cases2 cases

1 to 2 of 2

1. IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errors

   Shanghai AI Laboratory2024

   Symptoms

   Over six months from March to August 2023, LLM pretraining jobs on the Seren and Kalos clusters of Shanghai AI Laboratory's Acme datacenter, 4,704 A100 GPUs in total, failed again and again partway through and had to be restarted from the last checkpoint. Jobs that failed on infrastructure problems were only 11% of failed jobs by count but used over 82% of the GPU time of failed jobs.

   Training job failureGPU unavailableNCCL timeoutTraining divergence or loss spike

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLTraining data loaderObject storageData center power and cooling

   Contributing factor

   –

   Detection

   Failures were found in the framework's runtime logs, later by a diagnosis system that combines rules with an LLM; the pretraining framework detects loss spikes automatically. Jobs that stalled without an error were noticed only when users inspected them by hand.

   Remediation

   • Restarted interrupted pretraining jobs by hand from the last checkpoint, with developers on call in turn
   • Saved checkpoints at shorter intervals and let jobs terminate gracefully to keep their current state
   • Improved the cluster's cooling after GPU overheating caused NVLink and ECC errors
   • Found faulty nodes with a two-round NCCL allgather test and cordoned them off
   • Restarted from an earlier healthy checkpoint and skipped the following data batches after a loss spike
   • Fixed the garbage collection interval on each rank in InternEvo V2
   • Set num_worker=0 and enabled garbage collection in the dataloader to stop the memory leak

   Action items

   • Move failure diagnosis from GPT-4 to Shanghai AI Laboratory's own LLMs
   • Upgrade the infrastructure, with a particular focus on the NICs, and expand the computing cluster for larger-scale pretraining

   Lessons

   Infrastructure failures are rare by count but cost the most GPU time, so diagnosing them and recovering quickly matters most for training efficiency. Matching error logs with regular-expression rules alone is often inaccurate because the logs are diverse and one failure can print several errors. Jobs that stall without an error or that are paused for loss spikes waste resources until someone looks, which calls for a failure-handling system that finds and fixes faulty jobs by itself.

   View case
2. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts

   Linode2022

   Symptoms

   In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

   Intermittent timeoutsThroughput degradation

   Trigger

   One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

   Root cause

   Hardware failure

   Component

   Object storage

   Contributing factor

   Monitoring blind spotAutomation amplification

   Remediation

   • Removed the failed back-end infrastructure from the Object Storage cluster

   Action items

   • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
   • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly

   Lessons

   Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

   View case

By root cause1