Incident Library
Applied filters Root causeDependency failureFactorSingle point of failureOrganisationCrusoe CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1
    Crusoe Cloud2023
    Symptoms

    At 20:31 UTC on 15 November 2023, some GPU virtual machines in us-northcentral1-a became unavailable. One row of servers in a Crusoe Cloud data center in the us-northcentral1 region had lost power, and every server in the row powered off together with the VMs running on it.

    GPU unavailableService outage
    Trigger

    While the row's UPS was in bypass mode for routine maintenance, power switched from the primary source to a backup source for reasons unrelated to the maintenance.

    Root cause
    Dependency failure
    Component
    Data center power and cooling
    Contributing factor
    Single point of failureOverlapping failures
    Detection

    The power failure happened at 20:31 UTC. The first status post, at 21:07 UTC, said certain VMs in us-northcentral1-a were unavailable, and at 21:31 UTC the status page said the affected VMs had been identified.

    Remediation
    • identified the VMs on the row that lost power
    • restarted the affected VMs after power was restored
    Action items
    • Review the standard operating procedures for UPS maintenance to minimise customer impact during future maintenance
    • Build future data centers with redundant UPSs
    • Improve recovery procedures to reduce mean time to resolution
    Lessons

    When the only UPS for a row is in bypass for maintenance, the row is exposed to any power event during that window, even one unrelated to the work. Crusoe Cloud's response is redundant UPSs in future data centers, a review of maintenance procedures, and faster recovery of VMs after power returns.

    View case
Check your cluster for this

This case involves Data center power and cooling. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
Data center power and cooling
Contributing factor
Overlapping failuresSingle point of failure
Organisation
Crusoe Cloud
  1. IL-0135 A power source switch during UPS maintenance cut power to a row of GPU servers and shut down Crusoe Cloud VMs in us-northcentral1
    Crusoe Cloud2023