Incident Library
Applied filters ComponentInfiniBand fabricSymptomStorage disconnected or stale mountsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMs
    Lambda2026sev1
    Symptoms

    At 21:06 UTC on 14 February 2026 every GPU node in Lambda's AUS01 data center rebooted at the same moment, without warning or plan, affecting multiple customers. The nodes came back online quickly, but the storage they used had disconnected, and customer workloads failed.

    GPU unavailableStorage disconnected or stale mounts
    Trigger

    A power event at the AUS01 data center caused a loss of power.

    Root cause
    Dependency failure
    Component
    Data center power and coolingInfiniBand fabricShared network filesystem
    Contributing factor
    –
    Remediation
    • restored the compute fabric and InfiniBand connectivity, bringing all spine switches and most leaf switches back online
    • restored part of the UPS redundancy
    • powered up all nodes in AUS01, completed physical checks and started the storage back up
    • restored Starbase and the other storage-dependent services in us-west-3
    • told affected customers to reboot their VMs, or the hypervisor if needed, to clear stale or empty NFS mounts
    Lessons

    A power loss that reboots the network spine and the shared storage together with the GPU nodes takes longer to recover than the nodes themselves. The GPU nodes were back quickly, but the fabric was restored first, storage was started only after physical checks, and some VMs were left with stale NFS mounts that customers were told to fix with a VM reboot or, if that was not enough, a hypervisor reboot.

    View case
Check your cluster for this

This case involves Data center power and cooling and InfiniBand fabric. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
Data center power and coolingInfiniBand fabricShared network filesystem
Contributing factor
–
Organisation
Lambda
  1. IL-0134 A power event at Lambda's AUS01 data center rebooted 246 GPU nodes, all spine switches and the VAST storage, failing customer workloads and leaving stale NFS mounts on VMs
    Lambda2026