IL-0424 Nebius · 2024-11-26ai-infra
A Nebius Compute API release wrongly saw VMs as missing and restarted 282 VMs in eu-north1, leaving 164 stuck for hours
The new release contained a flawed assumption about the order in which events are processed by the system that tracks resource updates. With it, the Nebius Compute API wrongly concluded that some VMs were not running, marked them as missing and started recovery operations that restarted 282 VMs. The systems were not yet optimised for that many recovery operations at once, so many VMs got stuck. After the alert at 14:03, the team declared an incident within 15 minutes, identified the root cause at 15:00, halted all automatic recovery operations, separated the VMs that had restarted from those that were stuck, and tested a procedure on individual VMs: stop the VM to interrupt all stuck processes, then start it again in controlled batches. 163 VMs were recovered in batches between 15:37 and 16:39, and at 17:40 every affected VM was verified as started. Nebius plans more testing of recovery scenarios and stricter validation of VM state changes before deployment, rate limiting for recovery processes, better monitoring and anomaly detection of VM activity, extra safety checks in deployments, and faster escalation and better customer notification for large VM events.
- Severity
- –
- Scope
- 282 VMs in eu-north1 restarted, 196 got stuck, and 164 stayed offline for about 2 to 3 hours, interrupting customer workloads
- Users affected
- –
- Region
- eu-north1
- Time to detect
- 15m
- Time to resolve
- 3h 52m
Symptoms
On 26 November 2024 at 13:48 (UTC+1), right after a new release was deployed to Nebius's eu-north1 region, 282 virtual machines restarted at once and customer workloads on them were interrupted. 86 of them came back on their own within two minutes, but 196 got stuck and could not restart.
At 14:03 internal monitoring alerted on the large number of stuck VMs and phoned the on-call engineers, and the first customer report arrived over Slack at the same time. While the team worked out a safe way to recover, 32 VMs were restarted or deleted by their users; the remaining 164 stayed offline for about 2 to 3 hours until they were recovered by hand in batches.
Root cause
Root cause Faulty deployment
- Trigger
- Deployment of a new release, with a faulty change in the VM recovery sequence, to eu-north1 at 13:48 (UTC+1).
Detection
Internal monitoring alerted on the large number of stuck VMs at 14:03 and phoned the on-call engineers; the first customer report came over Slack at the same time.
Remediation
- Halted all automatic recovery operations to prevent further impact
- Split the affected VMs into those that had restarted and those stuck and needing manual work
- Tested a mitigation procedure on individual VMs before applying it
- Stopped the stuck VMs to interrupt their stuck processes and restarted them in controlled batches
Action items
- Add testing for recovery scenarios before deployment
- Enforce stricter validation of VM state changes
- Rate-limit recovery processes
- Improve monitoring and anomaly detection of VM activity
- Add safety checks to deployment procedures
- Escalate large-scale VM events faster
- Improve customer notifications and engagement, and refine internal communication protocols for incident response
Lessons
The staged manual recovery worked but needed close monitoring and hands-on work, which showed that better automation of recovery procedures would shorten the response to similar incidents.
Sources
Related cases
–
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0424
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0424/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.