IL-0135 Crusoe Cloud · 2023-11-15ai-infra
Crusoe Cloud was carrying out routine maintenance on the UPS serving one row of servers in a us-northcentral1 data center. To allow work on the UPS without interrupting power to the row, the UPS was put into bypass mode. During the maintenance window, power switched from the primary source to a backup source for unrelated reasons. With the UPS bypassed, nothing regulated the switchover, so the row lost power and all its servers shut down, stopping a percentage of the region's VMs.
Teams identified the affected VMs and restarted them as soon as power was restored. Service came back in stages, and the final VM was restarted at 00:20 UTC on 16 November 2023, 3 hours 49 minutes after the power failure. Crusoe Cloud said its future data centers will have redundant UPSs so this situation cannot recur, and that it is reviewing its UPS maintenance procedures.
At 20:31 UTC on 15 November 2023, some GPU virtual machines in us-northcentral1-a became unavailable. One row of servers in a Crusoe Cloud data center in the us-northcentral1 region had lost power, and every server in the row powered off together with the VMs running on it.
The status page reported certain VMs in us-northcentral1-a as unavailable at 21:07 UTC. The team identified the affected VMs and started restarting them once power was back; by 23:05 UTC most were running again, and the last VM was restarted at 00:20 UTC on 16 November.
Symptom GPU unavailableService outage
Root cause Dependency failure
The power failure happened at 20:31 UTC. The first status post, at 21:07 UTC, said certain VMs in us-northcentral1-a were unavailable, and at 21:31 UTC the status page said the affected VMs had been identified.
When the only UPS for a row is in bypass for maintenance, the row is exposed to any power event during that window, even one unrelated to the work. Crusoe Cloud's response is redundant UPSs in future data centers, a review of maintenance procedures, and faster recovery of VMs after power returns.
IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returnedNebius · 2026same root cause · shared Data center power and cooling
IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMsLambda · 2026 · sev1same root cause · shared Data center power and cooling
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.