Incident Library
Applied filters Root causeSoftware defectComponentBGP routingSymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
    Symptoms

    On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

    Network unreachable
    Trigger

    A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routing
    Contributing factor
    Ineffective rollbackGap in automated testsMonitoring blind spot
    Remediation
    • Restarted the host VPC component on the affected nodes one node at a time
    • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem
    Action items
    • Fix the bug in the VPC host component itself
    • Add an alert on a diverging number of routes
    • Add an alert on the number of drops in the data plane
    • Develop a way to reset endpoints without restarting the host VPC component
    • Move the VPC's persistent e2e test virtual machines to canary hosts
    • Add persistent e2e tests for Managed Services for Kubernetes
    • Run e2e tests more often
    • Add a deployment stage in which non-critical internal services receive updates first
    Lessons

    The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

    View case
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
    Symptoms

    On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

    Network unreachablePacket lossNode NotReady
    Trigger

    At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API
    Contributing factor
    Excessive blast radiusAutomation amplification
    Detection

    Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

    Remediation
    • Fixed both VPC route reflectors by manual action
    Action items
    • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
    • Run different software versions on the route reflectors so a single bug cannot stop several instances
    • Investigate the software issue that left the route reflectors in a stale state
    View case
  3. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
    Symptoms

    From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

    Network unreachableIntermittent timeoutsHigh latency
    Trigger

    The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

    Root cause
    Software defect
    Component
    BGP routing
    Contributing factor
    Excessive blast radiusHarmful manual action
    Remediation
    • Rolled out a software update to the edge routers running the affected version, site by site
    • Rebalanced traffic on the congested routes between AMS3 and FRA1
    • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug
    Action items
    • Upgrade all global edge routers to remove the bug
    • Bring the network upgrades already done in Asia to Europe and North America
    View case
  4. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019
    Symptoms

    On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

    Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage
    Trigger

    At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingBackbone network (WAN)
    Contributing factor
    Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback
    Detection

    Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

    Remediation
    • Lowered the priority of less critical network traffic classes to reduce congestion
    • Halted the datacenter automation software responsible for the maintenance event
    • Rescheduled the network control plane and its supporting infrastructure in each location
    • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
    • Directed service traffic away from the affected regions
    • Moved the Cloud VPN control plane in three regions out of its fail-safe state
    Action items
    • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
    • Harden the cluster management software to reject such requests whatever their origin
    • Reconfigure the network control plane to reject maintenance requests of the type involved
    • Make the network control plane in each location persist its configuration
    • Let the network stay in fail static mode longer when the control plane is lost
    • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
    • Extend continuous disaster recovery testing to this and similarly catastrophic failures
    View case
Check your cluster for this

These 4 cases involve BGP routing and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect4 cases

Component
BGP routing4Network control plane (SDN)3Backbone network (WAN)1Database1Managed Kubernetes control plane and provider API1Network switch1
Contributing factor
Excessive blast radius3Ineffective rollback2Automation amplification1Circular dependency1Limited debug access1Harmful manual action1Monitoring blind spot1Gap in automated tests1
Organisation
DigitalOcean1Google Cloud1Nebius1Scaleway1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  3. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
  4. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019