Cases3 cases

1 to 3 of 3

1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned

   Nebius2026

   Symptoms

   On 19 August 2026 the first equipment-level signs in Nebius's us-central1 region appeared around 09:40 UTC: the InfiniBand fabric degraded and GPU over-temperature alerts began to accumulate. Customer reports followed shortly after 10:00. At 10:15 on-call engineers were paged on critical component temperature alerts, and the incident was declared at 10:21.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A storm-related event at the data center building took the building management system offline and shut down the chilled-water cooling loop.

   Root cause

   Dependency failure

   Component

   Data center power and coolingCloud block storageetcdInfiniBand fabric

   Contributing factor

   Single point of failureMonitoring blind spotNo alert on limitNo runbookOverlapping failures

   Detection

   No facility-level alert fired, because the storm also disabled the building management system that monitors the facility. Engineers acted on component over-temperature alerts that paged on-call at 10:15 UTC, after InfiniBand degradation, GPU over-temperature alerts and customer reports; the temperature rise from about 09:15 was visible on dashboards but had no alert.

   Remediation

   • had the facility operator switch cooling to manual bypass mode and opened rack rear doors to vent heat
   • re-energised tripped racks, control-plane racks first
   • restored the regional infrastructure control plane by manual intervention
   • restarted block-storage host components to clear stale disk attachment state
   • raised the VM recovery rate limits by hand for the region-scale recovery
   • started the affected Managed Kubernetes control-plane instances by hand
   • applied a controlled stop and start to control-plane instances whose data disk had been left unattached
   • identified and started 104 customer VMs that the incident had stopped

   Action items

   • Complete a joint resilience review with the facility operator covering redundancy of facility-management and monitoring systems, cooling that keeps running when those fail, electrical protection of cooling equipment and on-site response
   • Add facility-level environmental alerting on cooling plant state and room temperature trend that pages on-call independently of the building management system
   • Set up a documented emergency escalation path with the facility operator, with an on-call contact tree and guaranteed response times
   • Define an emergency heat-load reduction procedure with criteria, ownership and mechanics for shutting down or degrading workloads during a cooling loss
   • Establish a dependency-ordered region cold-start procedure with clear ownership for each service layer
   • Run regular region-scale recovery drills that cover VMs, storage attachments and Managed Kubernetes clusters
   • Fix the disk-attachment bug that extended the Managed Kubernetes outage and deploy the fix to all production regions
   • Improve the recovery policies Compute users can configure, and move all Managed Kubernetes control-plane nodes to the improved policy
   • Automate reconciliation of stale storage attachment state after abrupt host failures
   • Remove in-region dependencies from monitoring startup and recovery, and extend cross-region monitoring
   • Make regional health checks used for global traffic management reflect real service availability

   Lessons

   Facility monitoring that runs on the same system it watches goes silent in exactly the failure it should report, and with no alert on the room temperature trend there was no practical window to shed heat load in a controlled way. A cold start of a whole region exercised recovery paths built for isolated failures: single-attempt VM recovery, conservative rate limits and stale disk attachments all needed manual work. The longest part of the outage was restoring customer resources, Managed Kubernetes control planes above all, so Nebius is extending its recovery drills to that layer.

   View case
2. IL-0023 A training run crashed with an NCCL timeout after one InfiniBand port went down once and recovered

   Databricks2026

   Symptoms

   Seven hours into a distributed training run on Databricks AI, the job failed with an NCCL timeout.

   Training job failureNCCL timeoutLink flapping

   Trigger

   A single InfiniBand port used for RDMA NCCL collectives went down once and recovered.

   Root cause

   Dependency failure

   Component

   NCCLInfiniBand fabricPyTorch

   Contributing factor

   Monitoring blind spotUnreviewed defaults

   Detection

   The training job crashed with the NCCL timeout. The continuous health checks, which watched InfiniBand port flapping by count, had not flagged the single flap.

   Remediation

   • Tuned the NCCL_IB_TIMEOUT defaults to be more resilient to short port-down windows
   • Used the port-down signal to crash the job and restart it from a checkpoint instead of waiting for the watchdog
   • Fed the investigation into gpu-monitor, whose RDMA fabric port-down check is thresholded on cumulative downtime rather than flap count

   Action items

   • Tune the NCCL_IB_TIMEOUT defaults to be more resilient
   • Crash and restart the job from a checkpoint on the port-down signal instead of leaving GPUs idle until the watchdog fires
   • Threshold RDMA fabric port-down detection on cumulative downtime, not flap count, in gpu-monitor

   Lessons

   For training impact the signal that matters is cumulative port downtime, not the number of flaps: one long enough flap can crash a multi-day run. NCCL_IB_TIMEOUT, with an effective default of about seven seconds, decides the outcome long before the PyTorch watchdog timeout does. Health checks need to cover the whole node lifecycle: active checks before workloads start, passive checks for silent degradation under load, and periodic multi-node NCCL fabric checks in between.

   View case
3. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training cluster

   Imbue2024

   Symptoms

   When Imbue brought up a cluster of 4,088 H100 GPUs on 511 hosts, about 10% of the machines failed to boot, mostly from physical problems: unconnected or miswired Ethernet cables, iDRAC hardware issues, broken power supplies, bad NVMe drives, missing internal wires, and network cards or GPUs that did not show up. In single-node GPU tests, Ubuntu server logs showed links between the GPUs and the PCIe buses or network cards reporting "limited width: x4 < x16". A few hosts had NVMe drives that locked up the whole machine when touched, drives that appeared in random order and got the OS installed on the wrong one, wrong temperature readings that kept fans at 100%, and CPUs limited to 2 GHz.

   Link flappingNCCL timeoutThroughput degradationGPU unavailableTraining job failure

   Trigger

   Bringing a new InfiniBand-connected H100 cluster into service and running the first multi-node training jobs on it.

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNCCLGPU driverPyTorch

   Contributing factor

   Unverified assumptionMonitoring blind spotPartial rollout

   Detection

   Automated hardware checks during provisioning, UFM alerts on temperature, error rates and flapping ports, Prometheus node exporter and DCGM metrics, per-batch MFU monitoring, and live debugging of stalled processes with Py-Spy, GDB and NVIDIA Nsight.

   Remediation

   • Automated checks for broken machines and sent faulty ones back to Dell or to data center staff
   • Reseated GPUs and internal PCIe cables and applied firmware upgrades
   • Downgraded the NVIDIA driver that caused wrong temperature readings
   • Rewired the top tier of the InfiniBand fabric into a single network and added checks that the wiring matched the design
   • Cleaned and reseated alerting ports and disabled the remaining alerting transceivers
   • Ran a burn-in workload through every InfiniBand port and disabled the ports that did not survive it
   • Enabled GPUDirect RDMA with an extra kernel module and PCIe ACS disabled
   • Disabled automatic garbage collection and scheduled it at set intervals on all hosts

   Action items

   • Release host-level health checks that confirm a host is free from known errors
   • Release an NCCL patch that improves logging around errors and stalls
   • Release a GPU stress test and NVLink and InfiniBand networking tests
   • Release a script that parses the UFM event log and decides which network ports to disable
   • Release a script that generates a burn-in workload exercising every InfiniBand link
   • Relaunch crashed runs automatically from the last checkpoint on the machines that pass health checks
   • Replace the disabled alerting transceivers

   Lessons

   Keep 10 to 20% more machines than a run needs, and network them so any working subset can be used. On the newest hardware expect about 3% of machines to break each week, mostly the same few machines breaking in different ways, so growing a set of known reliable "golden" hosts works better than chasing random failures. Write tests and automation for every kind of failure met, because each one comes back, and write tools that make opaque errors readable. Change only one thing at a time, and double-check the claims of new tools and people before building on them.

   View case

By root cause1