Incident Library

IL-0233 Nebius · 2025-03-13kubernetesnetworking

A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity

Pods of Nebius Managed Services for Kubernetes depend on the IP alias (prefix delegation) mechanism of virtual machine network interfaces. A host component of the VPC control plane sends, receives and processes BGP messages and announces and withdraws routes. A bug in that component, in production since a much earlier release and unknown to the developers, made it partly ignore route distinguishers, so some IP alias announcements that should have been kept apart were merged into one, and one withdrawal could cancel two different announcements. The release on 13 March changed route distinguisher values for a new feature, and only the update messages of Managed Kubernetes pods changed. A sequence of two announcements and one withdrawal then left the data plane with no active route, and packets for those IP aliases were dropped.

The team noticed the problem at 15:13 UTC and started a rollback at 15:30 UTC, but rolling back produced the same sequence of update messages with the route distinguishers reversed, so it did not help. At 16:41 UTC the team found that restarting the host VPC component on one node fixed that node and began restarting it on every affected node one at a time. The root cause was found at 17:36 UTC; once the team was sure the restart alone fixed the problem, it restarted the component on the remaining nodes in parallel from 17:57 UTC, and the impact was fully mitigated at 18:11 UTC. The bug needed two conditions to show: an IP alias had to exist when the release changed route distinguishers, and the node with the virtual machine sending to that IP alias had to be updated before the node with the receiving virtual machine. The permanent e2e test virtual machines on the testing installation met only the first condition, and none of them ran on canary nodes.

Severity
–
Scope
Practically every pod of Managed Services for Kubernetes in eu-west1 lost connectivity for at least some time between 13:36 and 18:11 UTC
Users affected
Clients running Managed Services for Kubernetes in eu-west1
Region
eu-west1
Time to detect
1h 37m
Time to resolve
4h 35m

Symptoms

On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

The Nebius team became aware of the problem at 15:13 UTC, about an hour and a half after the canary deployment of a VPC release started. Rolling the release back at 15:30 UTC did not help, which was clear by 16:00 UTC. At 16:20 UTC the team tied the problem to IP aliases without yet understanding it, and at 16:41 UTC found that restarting the VPC host component on a problematic node fixed that node. Two technical metrics, the number of routes and the number of drops, reacted to the problem, but alerting on technical metrics deviating from their normal values was insufficient.

Symptom Network unreachable

Root cause

Root cause Software defect

Component Network control plane (SDN)BGP routing
Contributing factor Ineffective rollbackGap in automated testsMonitoring blind spot
Trigger
A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

Detection

–

Remediation

Action items

Lessons

The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Network control plane (SDN), BGP routing among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.