Incident Library
Applied filters Root causeSoftware defectSymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

1 to 5 of 7

  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
    Symptoms

    On 13 March 2025 practically every pod of Nebius Managed Services for Kubernetes in the eu-west1 region lost network connectivity for at least some time between 13:36 and 18:11 UTC. Packets for the affected IP alias routes were not processed by the data plane. The eu-north1 region was not affected.

    Network unreachable
    Trigger

    A release of a VPC control plane component in eu-west1, deployed to canaries from 13:36 UTC and to all other nodes from 14:35 UTC, changed route distinguisher values for a new feature. The change made a long-standing bug in the handling of route announcements take effect for the update messages of Managed Kubernetes pods.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routing
    Contributing factor
    Ineffective rollbackGap in automated testsMonitoring blind spot
    Remediation
    • Restarted the host VPC component on the affected nodes one node at a time
    • Restarted the host VPC component on the remaining nodes in parallel once the restart was known to fix the problem
    Action items
    • Fix the bug in the VPC host component itself
    • Add an alert on a diverging number of routes
    • Add an alert on the number of drops in the data plane
    • Develop a way to reset endpoints without restarting the host VPC component
    • Move the VPC's persistent e2e test virtual machines to canary hosts
    • Add persistent e2e tests for Managed Services for Kubernetes
    • Run e2e tests more often
    • Add a deployment stage in which non-critical internal services receive updates first
    Lessons

    The failure needed both an existing IP alias at release time and a particular order of node updates, and neither the testing installation nor the canaries met both, so neither caught it. Missing alerts on technical metrics that deviate from normal, here the number of routes and the number of drops, made the response much slower.

    View case
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
    Symptoms

    On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.

    Network unreachablePacket lossNode NotReady
    Trigger

    At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingNetwork switchDatabaseManaged Kubernetes control plane and provider API
    Contributing factor
    Excessive blast radiusAutomation amplification
    Detection

    Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.

    Remediation
    • Fixed both VPC route reflectors by manual action
    Action items
    • Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
    • Run different software versions on the route reflectors so a single bug cannot stop several instances
    • Investigate the software issue that left the route reflectors in a stale state
    View case
  3. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped
    Google Cloud2024
    Symptoms

    From 15:22 US/Pacific on 16 May 2024, new VMs and other new instances across Google Cloud came up without network programming, so the new instances could not open network connections, and VMs that were migrated or restarted lost connectivity. Changes to virtual network configuration, such as firewall rules and load balancers, were not applied. Internal monitoring alerted Google engineers at 15:41.

    Network unreachablePacket lossCluster or node provisioning stallScale-up failureHigh error rate
    Trigger

    Google's infrastructure team ran a routine maintenance action meant to shut down one unused VPC Controller in a single Google Cloud zone, and the automation shut down VPC controllers in all zones where the controllers were still in use.

    Root cause
    Software defect
    Component
    Network control plane (SDN)Cloud load balancerNAT gateway
    Contributing factor
    Change with a delayed effectIgnored early signalNo safeguard on destructive changeExcessive blast radius
    Detection

    Internal monitoring alerts reached Google engineers at 15:41 US/Pacific, and the investigation started immediately.

    Remediation
    • Paused maintenance operations to prevent the shutdown from repeating
    • Brought the affected VPC Controllers back into a serving state in every zone
    • Stopped the automation responsible for the shutdown until safeguards are in place
    • Recovered partial Apigee proxy deployments by hand
    • Cleaned up VMs left behind by failed Dataproc cluster deletions
    Action items
    • Add safety measures that check that maintenance operations carry correctly scoped zone information
    • Make the underlying tools reject maintenance operations that do not specify zones
    • Tighten on-demand access control so one maintenance invocation cannot work across multiple zones
    • Add safety checks, testing and validation for maintenance operations, including positive and negative test cases for the new safety measures
    View case
  4. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
    Symptoms

    From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.

    Network unreachableIntermittent timeoutsHigh latency
    Trigger

    The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.

    Root cause
    Software defect
    Component
    BGP routing
    Contributing factor
    Excessive blast radiusHarmful manual action
    Remediation
    • Rolled out a software update to the edge routers running the affected version, site by site
    • Rebalanced traffic on the congested routes between AMS3 and FRA1
    • Rolled out the fix to the redundant networking equipment so a failover would not hit the bug
    Action items
    • Upgrade all global edge routers to remove the bug
    • Bring the network upgrades already done in Asia to Europe and North America
    View case
  5. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed
    Linode2023
    Symptoms

    On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

    Network unreachable
    Trigger

    A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

    Root cause
    Software defect
    Component
    Backbone network (WAN)NIC, cable and switch port link
    Contributing factor
    Overlapping failuresHarmful manual action
    Detection

    The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

    Remediation
    • Reverted the change that removed internet traffic from the Santa Clara border routers
    • Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
    • Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy
    View case
Check your cluster for this

These 7 cases involve Network control plane (SDN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect7 cases

Component
Network control plane (SDN)5BGP routing4Backbone network (WAN)2Cloud load balancer2Database1Managed Kubernetes control plane and provider API1NAT gateway1NIC, cable and switch port link1Network switch1
Contributing factor
Excessive blast radius5Harmful manual action2Ineffective rollback2Gap in automated tests2Automation amplification1Circular dependency1Change with a delayed effect1Ignored early signal1Limited debug access1Monitoring blind spot1No safeguard on destructive change1Overlapping failures1
Organisation
Google Cloud3DigitalOcean1Linode1Nebius1Scaleway1
  1. IL-0233 A VPC control plane release broke IP alias routes in Nebius eu-west1, and Managed Kubernetes pods lost connectivity
    Nebius2025
  2. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  3. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped
    Google Cloud2024
  4. IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
    DigitalOcean2023
  5. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed
    Linode2023
  6. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019
  7. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h
    Google Cloud2017