Cases3 cases

1 to 3 of 3

1. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API

   OpenAI2025

   Symptoms

   At 11:36 PM PDT on 9 June, alerts fired across multiple OpenAI services as a large number of cloud-hosted GPU nodes lost network connectivity, and the engineering team was escalated to immediately. Losing those nodes cut the capacity available to serve ChatGPT and the API.

   High error rateGPU unavailable

   Trigger

   A routine daily scheduled update of the host operating system on the cloud-hosted GPU servers restarted systemd-networkd on the affected nodes.

   Root cause

   Breaking change in an upgrade

   Component

   systemd-networkdNode operating system image

   Contributing factor

   No staged rolloutLimited debug access

   Detection

   Alerts fired immediately across multiple services when the infrastructure began losing network connectivity at 11:36 PM PDT, and the engineering team was escalated to at once.

   Remediation

   • re-imaged the affected GPU virtual machines
   • deployed recovery automation to restart the remaining impacted nodes
   • halted the background update mechanisms on the GPU VMs

   Action items

   • Disable automatic daily updates on GPU VMs so updates run in a controlled, scheduled way
   • Update system configurations to prevent conflicts between systemd-networkd and the networking service
   • Audit VM configurations across the fleet to find and reduce similar risks
   • Improve recovery speed for critical infrastructure such as GPU VMs and clusters
   • Run regular disaster recovery drills

   Lessons

   An unattended operating system update that restarts systemd-networkd can wipe the routes another networking agent manages, and it hits every node that takes the update at about the same time. Without break-glass tooling to restore connectivity, nodes that lost their network had to be re-imaged, which stretched recovery over many hours.

   View case
2. IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted it

   BigScience2022

   Symptoms

   During BigScience's 176B-parameter multilingual training (tr11-176B-ml) on the Jean Zay cluster, launched on 2022-03-11, one of the GPUs died about once a week. Most of the time the SLURM job crashed and restarted automatically from the last checkpoint.

   Training job failureGPU unavailable

   Trigger

   A GPU on one of the training nodes failed during the run.

   Root cause

   Hardware failure

   Component

   NCCLPyTorchSlurm job scheduler

   Contributing factor

   Unreviewed defaultsLimited debug access

   Detection

   There was no automatic detection of a hung job; it was noticed only when someone watching the training saw that it was stuck.

   Remediation

   • raised checkpoint frequency to every 200 iterations and then to every 100 iterations (SAVE_INTERVAL=100)
   • set NCCL_ASYNC_ERROR_HANDLING=1 so NCCL problems crash the job instead of hanging it
   • added the srun options --wait=60 and --kill-on-bad-exit=1 after testing them on 2 nodes

   Lessons

   On a long run across hundreds of GPUs, a dead GPU every week is routine; what costs time is the case where a process crashes and the job hangs instead of exiting, because then nothing restarts it. Making NCCL errors fatal and having the scheduler end the whole step on a bad exit is meant to turn such a hang into a crash that the existing automatic restart handles; only the srun part was tested. Frequent checkpoints bound the loss per crash, at the price of save time for a 2.3TB checkpoint.

   View case
3. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung

   Meta AI2021

   Symptoms

   Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.

   Training job failureGPU unavailable

   Trigger

   Nodes in the cloud training cluster failed with ECC errors, and each failure took the training run down.

   Root cause

   Dependency failure

   Component

   GPU hardware and memoryNCCLInfiniBand fabricObject storage

   Contributing factor

   No headroomLimited debug accessNo runbook

   Detection

   During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.

   Remediation

   • replaced failed nodes, and later did it through a script
   • added GPU burn-in testing to detect memory errors
   • automated InfiniBand testing
   • added monitoring of train.log
   • debugged the cloud provider's Python API until checkpoint downloads from the blob store worked well enough

   Action items

   • Run an on-call rotation for the training run, with runbooks and tooling for the most common issues, and invite more people to join it

   Lessons

   At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back. The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.

   View case

By root cause3