IL-0126 Meta AI · 2021-12-03 (published)ai-infra
Meta AI trained the 175B-parameter OPT model on GPU nodes rented from a cloud provider. In the two weeks after its 17 November 2021 update, the team restarted the run more than 40 times. The large majority of restarts were caused by hardware: nodes failed with ECC errors, and the team could not provision enough buffer nodes to replace a bad node quickly. A replacement through the cloud interface could take hours, often returned the same faulty machine, and sometimes arrived with NCCL or InfiniBand issues or the same ECC errors.
Recovery after each failure was slowed further by the checkpoint. A restart downloaded 1.6 TB (992 files of about 1.7 GB) from blob storage, and those downloads hung at random. Getting them to work reliably took considerable debugging of the provider's poorly documented Python API before the team settled on a solution it considered sufficient for the time being.
The team knew such failures would come at this scale but launched without automation, because it had to ship a trained 175B model by the end of H2 2021. Over the following two weeks it built its own tooling: a script to replace nodes, GPU burn-in tests to detect memory errors, automated InfiniBand tests, and monitoring of train.log. Working around infrastructure problems took up most of the team's time in that period, and the team set up an on-call rotation with runbooks for the most common issues.
Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.
Most restarts came from nodes that went down with ECC errors. The team could not get enough spare "buffer" nodes to swap in, and replacing one machine through the cloud provider's interface could take hours. More often than not the replacement turned out to be the same bad machine, and new nodes also came up with NCCL or InfiniBand problems or with the same ECC errors.
Each restart had to download the latest checkpoint, 1.6 TB in 992 files of about 1.7 GB each, from the cloud blob store, and these downloads started hanging nondeterministically, which delayed recovery further. The cloud provider's Python API for the blob store was poorly documented and needed a lot of debugging.
Root cause Dependency failure
During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.
At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back.
The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.
IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026same root cause · shared InfiniBand fabric, NCCL
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024same root cause · shared InfiniBand fabric, NCCL
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026same root cause · shared InfiniBand fabric
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1same root cause · shared InfiniBand fabric
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.