Cases1 case

1 to 1 of 1

1. IL-0424 A Nebius Compute API release wrongly saw VMs as missing and restarted 282 VMs in eu-north1, leaving 164 stuck for hours

   Nebius2024

   Symptoms

   On 26 November 2024 at 13:48 (UTC+1), right after a new release was deployed to Nebius's eu-north1 region, 282 virtual machines restarted at once and customer workloads on them were interrupted. 86 of them came back on their own within two minutes, but 196 got stuck and could not restart.

   Unexpected VM restartsService outage

   Trigger

   Deployment of a new release, with a faulty change in the VM recovery sequence, to eu-north1 at 13:48 (UTC+1).

   Root cause

   Faulty deployment

   Component

   Cloud compute control plane

   Contributing factor

   Unverified assumptionAutomation amplification

   Detection

   Internal monitoring alerted on the large number of stuck VMs at 14:03 and phoned the on-call engineers; the first customer report came over Slack at the same time.

   Remediation

   • Halted all automatic recovery operations to prevent further impact
   • Split the affected VMs into those that had restarted and those stuck and needing manual work
   • Tested a mitigation procedure on individual VMs before applying it
   • Stopped the stuck VMs to interrupt their stuck processes and restarted them in controlled batches

   Action items

   • Add testing for recovery scenarios before deployment
   • Enforce stricter validation of VM state changes
   • Rate-limit recovery processes
   • Improve monitoring and anomaly detection of VM activity
   • Add safety checks to deployment procedures
   • Escalate large-scale VM events faster
   • Improve customer notifications and engagement, and refine internal communication protocols for incident response

   Lessons

   The staged manual recovery worked but needed close monitoring and hands-on work, which showed that better automation of recovery procedures would shorten the response to similar incidents.

   View case

By root cause1