Cases3 cases

1 to 3 of 3

1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned

   Nebius2026

   Symptoms

   On 19 August 2026 the first equipment-level signs in Nebius's us-central1 region appeared around 09:40 UTC: the InfiniBand fabric degraded and GPU over-temperature alerts began to accumulate. Customer reports followed shortly after 10:00. At 10:15 on-call engineers were paged on critical component temperature alerts, and the incident was declared at 10:21.

   Control plane unresponsiveService outageMonitoring outage

   Trigger

   A storm-related event at the data center building took the building management system offline and shut down the chilled-water cooling loop.

   Root cause

   Dependency failure

   Component

   Data center power and coolingCloud block storageetcdInfiniBand fabric

   Contributing factor

   Single point of failureMonitoring blind spotNo alert on limitNo runbookOverlapping failures

   Detection

   No facility-level alert fired, because the storm also disabled the building management system that monitors the facility. Engineers acted on component over-temperature alerts that paged on-call at 10:15 UTC, after InfiniBand degradation, GPU over-temperature alerts and customer reports; the temperature rise from about 09:15 was visible on dashboards but had no alert.

   Remediation

   • had the facility operator switch cooling to manual bypass mode and opened rack rear doors to vent heat
   • re-energised tripped racks, control-plane racks first
   • restored the regional infrastructure control plane by manual intervention
   • restarted block-storage host components to clear stale disk attachment state
   • raised the VM recovery rate limits by hand for the region-scale recovery
   • started the affected Managed Kubernetes control-plane instances by hand
   • applied a controlled stop and start to control-plane instances whose data disk had been left unattached
   • identified and started 104 customer VMs that the incident had stopped

   Action items

   • Complete a joint resilience review with the facility operator covering redundancy of facility-management and monitoring systems, cooling that keeps running when those fail, electrical protection of cooling equipment and on-site response
   • Add facility-level environmental alerting on cooling plant state and room temperature trend that pages on-call independently of the building management system
   • Set up a documented emergency escalation path with the facility operator, with an on-call contact tree and guaranteed response times
   • Define an emergency heat-load reduction procedure with criteria, ownership and mechanics for shutting down or degrading workloads during a cooling loss
   • Establish a dependency-ordered region cold-start procedure with clear ownership for each service layer
   • Run regular region-scale recovery drills that cover VMs, storage attachments and Managed Kubernetes clusters
   • Fix the disk-attachment bug that extended the Managed Kubernetes outage and deploy the fix to all production regions
   • Improve the recovery policies Compute users can configure, and move all Managed Kubernetes control-plane nodes to the improved policy
   • Automate reconciliation of stale storage attachment state after abrupt host failures
   • Remove in-region dependencies from monitoring startup and recovery, and extend cross-region monitoring
   • Make regional health checks used for global traffic management reflect real service availability

   Lessons

   Facility monitoring that runs on the same system it watches goes silent in exactly the failure it should report, and with no alert on the room temperature trend there was no practical window to shed heat load in a controlled way. A cold start of a whole region exercised recovery paths built for isolated failures: single-attempt VM recovery, conservative rate limits and stale disk attachments all needed manual work. The longest part of the outage was restoring customer resources, Managed Kubernetes control planes above all, so Nebius is extending its recovery drills to that layer.

   View case
2. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down

   UK Ministry of Justice2025

   Symptoms

   At 09:36 on 18 March 2025, during the upgrade of the live cluster to EKS 1.30, users reported that their pods were being terminated and were not restarting automatically, in both production and non-production environments. Automated infrastructure alerts fired as well, and production services had outages.

   Pod evictionPods PendingService outageUpgrade stall

   Trigger

   Upgrading the live EKS cluster from 1.29 to 1.30, during which the descheduler attempted to reschedule all pods on the cordoned nodes simultaneously.

   Root cause

   Faulty deployment

   Component

   DeschedulerNode poolCluster autoscaler

   Contributing factor

   No headroomNo alert on limitUnreviewed defaults

   Detection

   Users reported at 09:36 that pods were being terminated and not restarting, and automated infrastructure alerts fired.

   Remediation

   • Removed the cordon annotation so pods could be rescheduled
   • Deleted stuck 1.30 nodes to force their recreation
   • Added a new node group of 20 nodes running 1.30 by hand
   • Cordoned and drained all old 1.29 nodes onto the new 1.30 nodes

   Action items

   • Raise an AWS Production System Down support case (Case ID: 174229643100496)
   • Get guidance from AWS support on querying control-plane logs to find the root cause of similar issues
   • Document a runbook for emergency scaling of a new node group in case of unexpected capacity shortages
   • Alert on an imbalance between the autoscaling group and the Kubernetes nodes
   • Review the descheduler configuration to understand why it throws pods off cordoned nodes, and fix it where possible
   • Alert on spikes in the pending pod count

   Lessons

   The write-up lists review actions rather than lessons. They show the team had no documented way to add an emergency node group, no alert on a mismatch between the autoscaling group and the nodes Kubernetes sees, no alert on a spike in pending pods, and no clear understanding of how the descheduler treats cordoned nodes.

   View case
3. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models

   OpenAI2023

   Symptoms

   Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.

   High error rateService outageProbe failures

   Trigger

   A morning with dramatically more completions than any previous day pushed the routing layer nodes, which already had a chronic memory problem, past their memory limits.

   Root cause

   Software defect

   Component

   Model request routingContainer memory limit

   Contributing factor

   No alert on limitNo headroom

   Remediation

   • limited incoming traffic
   • redeployed the routing service en masse
   • ramped traffic back up slowly

   Action items

   • Pre-allocate the responseBuffer and reuse it instead of allocating a new one in a loop (3X improvement in memory and CPU usage)
   • Adjust the configured memory limits to an appropriate level so the service has significant headroom
   • Add rate limit controls to shed traffic more gracefully
   • Increase the service's capacity
   • Change alerting to catch the underlying memory behaviour before it escalates into service issues
   • Configure autoscaling for the service, which was not possible before because added capacity could hurt an upstream service

   View case

By root cause3