Cases2 cases

1 to 2 of 2

1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned

   Nebius2026

   Symptoms

   On 19 August 2026 the first equipment-level signs in Nebius's us-central1 region appeared around 09:40 UTC: the InfiniBand fabric degraded and GPU over-temperature alerts began to accumulate. Customer reports followed shortly after 10:00. At 10:15 on-call engineers were paged on critical component temperature alerts, and the incident was declared at 10:21.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A storm-related event at the data center building took the building management system offline and shut down the chilled-water cooling loop.

   Root cause

   Dependency failure

   Component

   Data center power and coolingCloud block storageetcdInfiniBand fabric

   Contributing factor

   Single point of failureMonitoring blind spotNo alert on limitNo runbookOverlapping failures

   Detection

   No facility-level alert fired, because the storm also disabled the building management system that monitors the facility. Engineers acted on component over-temperature alerts that paged on-call at 10:15 UTC, after InfiniBand degradation, GPU over-temperature alerts and customer reports; the temperature rise from about 09:15 was visible on dashboards but had no alert.

   Remediation

   • had the facility operator switch cooling to manual bypass mode and opened rack rear doors to vent heat
   • re-energised tripped racks, control-plane racks first
   • restored the regional infrastructure control plane by manual intervention
   • restarted block-storage host components to clear stale disk attachment state
   • raised the VM recovery rate limits by hand for the region-scale recovery
   • started the affected Managed Kubernetes control-plane instances by hand
   • applied a controlled stop and start to control-plane instances whose data disk had been left unattached
   • identified and started 104 customer VMs that the incident had stopped

   Action items

   • Complete a joint resilience review with the facility operator covering redundancy of facility-management and monitoring systems, cooling that keeps running when those fail, electrical protection of cooling equipment and on-site response
   • Add facility-level environmental alerting on cooling plant state and room temperature trend that pages on-call independently of the building management system
   • Set up a documented emergency escalation path with the facility operator, with an on-call contact tree and guaranteed response times
   • Define an emergency heat-load reduction procedure with criteria, ownership and mechanics for shutting down or degrading workloads during a cooling loss
   • Establish a dependency-ordered region cold-start procedure with clear ownership for each service layer
   • Run regular region-scale recovery drills that cover VMs, storage attachments and Managed Kubernetes clusters
   • Fix the disk-attachment bug that extended the Managed Kubernetes outage and deploy the fix to all production regions
   • Improve the recovery policies Compute users can configure, and move all Managed Kubernetes control-plane nodes to the improved policy
   • Automate reconciliation of stale storage attachment state after abrupt host failures
   • Remove in-region dependencies from monitoring startup and recovery, and extend cross-region monitoring
   • Make regional health checks used for global traffic management reflect real service availability

   Lessons

   Facility monitoring that runs on the same system it watches goes silent in exactly the failure it should report, and with no alert on the room temperature trend there was no practical window to shed heat load in a controlled way. A cold start of a whole region exercised recovery paths built for isolated failures: single-attempt VM recovery, conservative rate limits and stale disk attachments all needed manual work. The longest part of the outage was restoring customer resources, Managed Kubernetes control planes above all, so Nebius is extending its recovery drills to that layer.

   View case
2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case

By root cause2