Cases3 cases

1 to 3 of 3

1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity

   Nebius2025

   Symptoms

   On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

   Network unreachable

   Trigger

   A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

   Root cause

   Software defect

   Component

   Network control plane (SDN)BGP routing

   Contributing factor

   Ineffective rollbackGap in automated testsMonitoring blind spot

   Remediation

   • Restarted the host VPC component on the affected nodes one node at a time
   • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem

   Action items

   • Fix the bug in the VPC host component itself
   • Add an alert on a diverging number of routes
   • Add an alert on the number of drops in the data plane
   • Develop a way to reset endpoints without restarting the host VPC component
   • Move the VPC's persistent e2e test virtual machines to canary hosts
   • Add persistent e2e tests for Managed Services for Kubernetes
   • Run e2e tests more often
   • Add a deployment stage in which non-critical internal services receive updates first

   Lessons

   The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

   View case
2. IL-0259 Outbound packets from Fastmail did not reach part of the internet for about 19 hours, cutting off 3-5% of customers

   Fastmail2023

   Symptoms

   From about 03:00 UTC on 30 June 2023, Fastmail and a fraction of the internet could not communicate with each other, and Fastmail first heard of it from customers through support tickets and social media. The problem lasted about 19 hours and affected an estimated 3-5% of customers as well as some email servers.

   Network unreachable

   Root cause

   Dependency failure

   Component

   ISP and transit networks

   Contributing factor

   Single point of failureMonitoring blind spot

   Detection

   Customers told Fastmail through support tickets and social media. Fastmail's own monitors in multiple locations and its staff were not on affected networks and saw nothing.

   Action items

   • Select a provider for a second transit connection directly into Fastmail's servers, via Megaport or a provider present in 365's New Jersey datacenter, so that Fastmail controls the outbound path and can route around a network with problems

   Lessons

   With a single path for outbound traffic, Fastmail could not route around a provider's network problem and could not fix it itself. Being able to choose between several outbound paths would let Fastmail get everyone back online faster.

   View case
3. IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance

   OpenStreetMap Foundation2022

   Symptoms

   From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.

   Network unreachableService outage

   Trigger

   Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.

   Root cause

   Hardware failure

   Component

   Network switchData center power and cooling

   Contributing factor

   Single point of failureLimited debug accessMonitoring blind spot

   Remediation

   • Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC

   Action items

   • Acknowledge ongoing outages faster
   • Add monitoring and alerting of switch PSU and FAN states
   • Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
   • Install out-of-band access equipment with dual power feeds in Amsterdam
   • Install redundant uplinks
   • Make site failover easier, less manual and lower risk
   • Introduce an on-call pager system for the operations SRE
   • Expand the SRE team
   • Continue and formalise fail-over tests for power, network, upstream link and service outages

   View case

By root cause3