Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
    Symptoms

    On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

    Network unreachable
    Trigger

    A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routing
    Contributing factor
    Ineffective rollbackGap in automated testsMonitoring blind spot
    Remediation
    • Restarted the host VPC component on the affected nodes one node at a time
    • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem
    Action items
    • Fix the bug in the VPC host component itself
    • Add an alert on a diverging number of routes
    • Add an alert on the number of drops in the data plane
    • Develop a way to reset endpoints without restarting the host VPC component
    • Move the VPC's persistent e2e test virtual machines to canary hosts
    • Add persistent e2e tests for Managed Services for Kubernetes
    • Run e2e tests more often
    • Add a deployment stage in which non-critical internal services receive updates first
    Lessons

    The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

    View case
Check your cluster for this

This case involves BGP routing and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
BGP routingNetwork control plane (SDN)
Contributing factor
Monitoring blind spotIneffective rollbackGap in automated tests
Organisation
Nebius
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025