Cases2 cases

1 to 2 of 2

1. IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMs

   Lambda2026sev1

   Symptoms

   At 21:06 UTC on 14 February 2026 every GPU node in Lambda's AUS01 data center rebooted at the same moment, without warning or plan, affecting multiple customers. The nodes came back online quickly, but the storage they used had disconnected, and customer workloads failed.

   GPU unavailableStorage disconnected or stale mounts

   Trigger

   A power event at the AUS01 data center caused a loss of power.

   Root cause

   Dependency failure

   Component

   Data center power and coolingInfiniBand fabricShared network filesystem

   Contributing factor

   –

   Remediation

   • restored the compute fabric and InfiniBand connectivity, bringing all spine switches and most leaf switches back online
   • restored part of the UPS redundancy
   • powered up all nodes in AUS01, completed physical checks and started the storage back up
   • restored Starbase and the other storage-dependent services in us-west-3
   • told affected customers to reboot their VMs, or the hypervisor if needed, to clear stale or empty NFS mounts

   Lessons

   A power loss that reboots the network spine and the shared storage together with the GPU nodes takes longer to recover than the nodes themselves. The GPU nodes were back quickly, but the fabric was restored first, storage was started only after physical checks, and some VMs were left with stale NFS mounts that customers were told to fix with a VM reboot or, if that was not enough, a hypervisor reboot.

   View case
2. IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1

   Crusoe Cloud2023

   Symptoms

   At 20:31 UTC on 15 November 2023, some GPU virtual machines in us-northcentral1-a became unavailable. One row of servers in a Crusoe Cloud data center in the us-northcentral1 region had lost power, and every server in the row powered off together with the VMs running on it.

   GPU unavailableService outage

   Trigger

   While the row's UPS was in bypass mode for routine maintenance, power switched from the primary source to a backup source for reasons unrelated to the maintenance.

   Root cause

   Dependency failure

   Component

   Data center power and cooling

   Contributing factor

   Single point of failureOverlapping failures

   Detection

   The power failure happened at 20:31 UTC. The first status post, at 21:07 UTC, said certain VMs in us-northcentral1-a were unavailable, and at 21:31 UTC the status page said the affected VMs had been identified.

   Remediation

   • identified the VMs on the row that lost power
   • restarted the affected VMs after power was restored

   Action items

   • Review the standard operating procedures for UPS maintenance to minimise customer impact during future maintenance
   • Build future data centers with redundant UPSs
   • Improve recovery procedures to reduce mean time to resolution

   Lessons

   When the only UPS for a row is in bypass for maintenance, the row is exposed to any power event during that window, even one unrelated to the work. Crusoe Cloud's response is redundant UPSs in future data centers, a review of maintenance procedures, and faster recovery of VMs after power returns.

   View case

By root cause1