Cases3 cases

1 to 3 of 3

1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned

   Nebius2026

   Symptoms

   On 19 August 2026 the first equipment-level signs in Nebius's us-central1 region appeared around 09:40 UTC: the InfiniBand fabric degraded and GPU over-temperature alerts began to accumulate. Customer reports followed shortly after 10:00. At 10:15 on-call engineers were paged on critical component temperature alerts, and the incident was declared at 10:21.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A storm-related event at the data center building took the building management system offline and shut down the chilled-water cooling loop.

   Root cause

   Dependency failure

   Component

   Data center power and coolingCloud block storageetcdInfiniBand fabric

   Contributing factor

   Single point of failureMonitoring blind spotNo alert on limitNo runbookOverlapping failures

   Detection

   No facility-level alert fired, because the storm also disabled the building management system that monitors the facility. Engineers acted on component over-temperature alerts that paged on-call at 10:15 UTC, after InfiniBand degradation, GPU over-temperature alerts and customer reports; the temperature rise from about 09:15 was visible on dashboards but had no alert.

   Remediation

   • had the facility operator switch cooling to manual bypass mode and opened rack rear doors to vent heat
   • re-energised tripped racks, control-plane racks first
   • restored the regional infrastructure control plane by manual intervention
   • restarted block-storage host components to clear stale disk attachment state
   • raised the VM recovery rate limits by hand for the region-scale recovery
   • started the affected Managed Kubernetes control-plane instances by hand
   • applied a controlled stop and start to control-plane instances whose data disk had been left unattached
   • identified and started 104 customer VMs that the incident had stopped

   Action items

   • Complete a joint resilience review with the facility operator covering redundancy of facility-management and monitoring systems, cooling that keeps running when those fail, electrical protection of cooling equipment and on-site response
   • Add facility-level environmental alerting on cooling plant state and room temperature trend that pages on-call independently of the building management system
   • Set up a documented emergency escalation path with the facility operator, with an on-call contact tree and guaranteed response times
   • Define an emergency heat-load reduction procedure with criteria, ownership and mechanics for shutting down or degrading workloads during a cooling loss
   • Establish a dependency-ordered region cold-start procedure with clear ownership for each service layer
   • Run regular region-scale recovery drills that cover VMs, storage attachments and Managed Kubernetes clusters
   • Fix the disk-attachment bug that extended the Managed Kubernetes outage and deploy the fix to all production regions
   • Improve the recovery policies Compute users can configure, and move all Managed Kubernetes control-plane nodes to the improved policy
   • Automate reconciliation of stale storage attachment state after abrupt host failures
   • Remove in-region dependencies from monitoring startup and recovery, and extend cross-region monitoring
   • Make regional health checks used for global traffic management reflect real service availability

   Lessons

   Facility monitoring that runs on the same system it watches goes silent in exactly the failure it should report, and with no alert on the room temperature trend there was no practical window to shed heat load in a controlled way. A cold start of a whole region exercised recovery paths built for isolated failures: single-attempt VM recovery, conservative rate limits and stale disk attachments all needed manual work. The longest part of the outage was restoring customer resources, Managed Kubernetes control planes above all, so Nebius is extending its recovery drills to that layer.

   View case
2. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case
3. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case

By root cause3