Cases1 case

1 to 1 of 1

1. IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMs

   Lambda2026sev1

   Symptoms

   At 21:06 UTC on 14 February 2026 every GPU node in Lambda's AUS01 data center rebooted at the same moment, without warning or plan, affecting multiple customers. The nodes came back online quickly, but the storage they used had disconnected, and customer workloads failed.

   GPU unavailableStorage disconnected or stale mounts

   Trigger

   A power event at the AUS01 data center caused a loss of power.

   Root cause

   Dependency failure

   Component

   Data center power and coolingInfiniBand fabricShared network filesystem

   Contributing factor

   –

   Remediation

   • restored the compute fabric and InfiniBand connectivity, bringing all spine switches and most leaf switches back online
   • restored part of the UPS redundancy
   • powered up all nodes in AUS01, completed physical checks and started the storage back up
   • restored Starbase and the other storage-dependent services in us-west-3
   • told affected customers to reboot their VMs, or the hypervisor if needed, to clear stale or empty NFS mounts

   Lessons

   A power loss that reboots the network spine and the shared storage together with the GPU nodes takes longer to recover than the nodes themselves. The GPU nodes were back quickly, but the fabric was restored first, storage was started only after physical checks, and some VMs were left with stale NFS mounts that customers were told to fix with a VM reboot or, if that was not enough, a hypervisor reboot.

   View case

By root cause1