Cases2 cases

1 to 2 of 2

1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity

   Nebius2025

   Symptoms

   On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

   Network unreachable

   Trigger

   A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routing

   Contributing factor

   Ineffective rollbackGap in automated testsMonitoring blind spot

   Remediation

   • Restarted the host VPC component on the affected nodes one node at a time
   • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem

   Action items

   • Fix the bug in the VPC host component itself
   • Add an alert on a diverging number of routes
   • Add an alert on the number of drops in the data plane
   • Develop a way to reset endpoints without restarting the host VPC component
   • Move the VPC's persistent e2e test virtual machines to canary hosts
   • Add persistent e2e tests for Managed Services for Kubernetes
   • Run e2e tests more often
   • Add a deployment stage in which non-critical internal services receive updates first

   Lessons

   The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

   View case
2. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage

   Monzo2017

   Symptoms

   At 14:10 an engineer deployed a change to a service that Monzo's current account needs to process payments, and every request to that service started to fail. Current account customers saw payment failures; the prepaid card was not affected at first because it does not use that service. The change was rolled back at 14:12, but the errors continued.

   High error rateStale endpoint routingService outage

   Trigger

   At 14:10 an engineer deployed a change to a service needed to process current account payments. Two weeks earlier the etcd cluster had been upgraded and grown from three to nine nodes, and one day earlier another team had scaled a new service down to zero replicas while leaving its Kubernetes service in place.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverLinkerdkubelet

   Contributing factor

   Ineffective rollbackHarmful manual actionOverlapping failuresMonitoring blind spotSingle point of failure

   Detection

   Payment failures began right after the 14:10 deploy. The team rolled back at 14:12 and declared an outage internally at 14:16. At 15:13, when no service could serve requests, practically all automated alerts fired.

   Remediation

   • Rolled back the payments service change (the errors continued)
   • Activated internal backup processes so most customers could still use their card
   • Restarted all linkerd instances (this made the outage worse)
   • Restarted the three Kubernetes API server processes so replacement linkerd instances could start
   • Began deploying a newer linkerd version with the fix for empty services
   • Deleted the Kubernetes service that had no endpoints, which let linkerd load service discovery again

   Action items

   • Fix the bug in Kubernetes that can trigger timeouts after a cluster reconfiguration
   • Roll out a new version of linkerd that fixes the parsing error
   • Create better health checks, dashboards and alerts for the affected components so they give clearer signals about what is wrong and prevent human error
   • Improve procedures to communicate outages internally and externally as clearly and quickly as possible

   Lessons

   A large failure in a distributed system can be very hard to understand, and well-meant human action can compound it, as restarting every linkerd instance did here by exposing a second incompatibility. Because Monzo deploys often and its network changes constantly, the deployment and request-routing systems have to be trustworthy. Monzo committed to publishing a similar post-mortem for any incident of this severity.

   View case

By root cause1