IL-0134 Lambda · 2026-02-14ai-infranetworking
A power event at Lambda's AUS01 data center cut power and rebooted 246 GPU nodes, all 4 spine switches, the mCPU nodes and the VAST storage infrastructure at 21:06 UTC on 14 February 2026. The GPU nodes came back quickly, but storage had disconnected and customer workloads failed.
Engineers restored the compute fabric and InfiniBand connectivity, bringing all spine switches and most leaf switches back, and restored part of the UPS redundancy while work continued on storage and protected power. They then confirmed that all nodes in AUS01 were powered up with network connectivity, completed physical checks, and moved on to the logical startup of storage, which NFS and the services that depend on it need. Starbase and the other storage-dependent services in us-west-3 were then restored.
When the incident was marked resolved, the main remaining problem was VMs with stale or empty NFS mounts. Lambda told affected customers to reboot their VMs to get file system access back, and to reboot the hypervisor if a VM reboot was not enough.
At 21:06 UTC on 14 February 2026 every GPU node in Lambda's AUS01 data center rebooted at the same moment, without warning or plan, affecting multiple customers. The nodes came back online quickly, but the storage they used had disconnected, and customer workloads failed.
The power loss had rebooted 246 GPU nodes, all 4 spine switches, the mCPU nodes and the VAST storage infrastructure. Lambda restored the compute fabric and InfiniBand connectivity first and started the storage again later.
After storage and storage-dependent services were back, some customer VMs still had stale or empty NFS mounts left over from the power event and the mass reboots.
Root cause Dependency failure
–
–
A power loss that reboots the network spine and the shared storage together with the GPU nodes takes longer to recover than the nodes themselves. The GPU nodes were back quickly, but the fabric was restored first, storage was started only after physical checks, and some VMs were left with stale NFS mounts that customers were told to fix with a VM reboot or, if that was not enough, a hypervisor reboot.
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026same root cause · shared Data center power and cooling, InfiniBand fabric
IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recoveredDatabricks · 2026same root cause · shared InfiniBand fabric
IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024same root cause · shared InfiniBand fabric
IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1Crusoe Cloud · 2023same root cause · shared Data center power and cooling
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared InfiniBand fabric