Cases2 cases

1 to 2 of 2

1. IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errors

   Shanghai AI Laboratory2024

   Symptoms

   Over six months from March to August 2023, LLM pretraining jobs on the Seren and Kalos clusters of Shanghai AI Laboratory's Acme datacenter, 4,704 A100 GPUs in total, failed again and again partway through and had to be restarted from the last checkpoint. Jobs that failed on infrastructure problems were only 11% of failed jobs by count but used over 82% of the GPU time of failed jobs.

   Training job failureGPU unavailableNCCL timeoutTraining divergence or loss spike

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLTraining data loaderObject storageData center power and cooling

   Contributing factor

   –

   Detection

   Failures were found in the framework's runtime logs, later by a diagnosis system that combines rules with an LLM; the pretraining framework detects loss spikes automatically. Jobs that stalled without an error were noticed only when users inspected them by hand.

   Remediation

   • Restarted interrupted pretraining jobs by hand from the last checkpoint, with developers on call in turn
   • Saved checkpoints at shorter intervals and let jobs terminate gracefully to keep their current state
   • Improved the cluster's cooling after GPU overheating caused NVLink and ECC errors
   • Found faulty nodes with a two-round NCCL allgather test and cordoned them off
   • Restarted from an earlier healthy checkpoint and skipped the following data batches after a loss spike
   • Fixed the garbage collection interval on each rank in InternEvo V2
   • Set num_worker=0 and enabled garbage collection in the dataloader to stop the memory leak

   Action items

   • Move failure diagnosis from GPT-4 to Shanghai AI Laboratory's own LLMs
   • Upgrade the infrastructure, with a particular focus on the NICs, and expand the computing cluster for larger-scale pretraining

   Lessons

   Infrastructure failures are rare by count but cost the most GPU time, so diagnosing them and recovering quickly matters most for training efficiency. Matching error logs with regular-expression rules alone is often inaccurate because the logs are diverse and one failure can print several errors. Jobs that stall without an error or that are paused for loss spikes waste resources until someone looks, which calls for a failure-handling system that finds and fixes faulty jobs by itself.

   View case
2. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

   OpenStreetMap Foundation2022

   Symptoms

   From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

   Network unreachableService outage

   Trigger

   Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

   Root cause

   Hardware failure

   Component

   Network switchData center power and cooling

   Contributing factor

   Single point of failureLimited debug accessMonitoring blind spot

   Remediation

   • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

   Action items

   • Acknowledge ongoing outages faster
   • Add monitoring and alerting of switch PSU and FAN states
   • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
   • Install out-of-band access equipment with dual power feeds in Amsterdam
   • Install redundant uplinks
   • Make site failover easier, less manual and lower risk
   • Introduce an on-call pager system for the operations SRE
   • Expand the SRE team
   • Continue and formalise fail-over tests for power, network, upstream link and service outages

   View case

By root cause1