Incident Library

Applied filters Root causeSoftware defectComponentBGP routingFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases3 cases

1 to 3 of 3

IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
Scaleway2024
Symptoms
On 18 October 2024, from 7:50 to 8:01 UTC, about 25% of the hypervisors in Scaleway's FR-PAR-1 availability zone lost VPC connectivity with the rest of the region, and products that use VPC failed with them. Public internet connectivity and Elastic Metal VPC connectivity were not affected.
Network unreachable Packet loss Node NotReady
Trigger
At 6:20 UTC a Top of Rack switch in FR-PAR-1 had a software crash. One of the two BGP route reflectors for VPC ran on a hypervisor in that rack and was left in an abnormal state.
Root cause
Software defect
Component
Network control plane (SDN)BGP routing Network switch Database Managed Kubernetes control plane and provider API
Contributing factor
Excessive blast radius Automation amplification
Detection
Health-check monitoring on both route reflectors detected the anomaly, and the Scaleway Customer Excellence and SRE teams detected the customer impact and opened an incident.
Remediation
Fixed both VPC route reflectors by manual action
Action items
Fixed the autohealing mechanism that had failed to fix route reflectors stuck in a stale state; the fix is https://github.com/FRRouting/frr/pull/17163
Run different software versions on the route reflectors so a single bug cannot stop several instances
Investigate the software issue that left the route reflectors in a stale state
View case
IL-0342 DigitalOcean's AMS3, LON1 and SFO2 data centers lost Internet connectivity in repeated waves as edge routers failed
DigitalOcean2023
Symptoms
From 09:55 UTC on 9 August 2023, DigitalOcean's AMS3, LON1 and SFO2 datacenters lost Internet access in waves of 10 to 15 minutes, about an hour apart. Users saw connectivity issues, latency and timeout errors on Droplets and on Droplet-based services such as Load Balancers, Kubernetes and Database clusters.
Network unreachable Intermittent timeouts High latency
Trigger
The edge routers received an RPKI-related routing update, and the operating system software on them crashed its routing process on that update.
Root cause
Software defect
Component
BGP routing
Contributing factor
Excessive blast radius Harmful manual action
Remediation
Rolled out a software update to the edge routers running the affected version, site by site
Rebalanced traffic on the congested routes between AMS3 and FRA1
Rolled out the fix to the redundant networking equipment so a failover would not hit the bug
Action items
Upgrade all global edge routers to remove the bug
Bring the network upgrades already done in Asia to Europe and North America
View case
IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
Google Cloud2019
Symptoms
On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.
Packet loss Network unreachable High latency High error rate Monitoring outage
Trigger
At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.
Root cause
Software defect
Component
Network control plane (SDN)BGP routing Backbone network (WAN)
Contributing factor
Excessive blast radius Circular dependency Limited debug access Ineffective rollback
Detection
Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.
Remediation
Lowered the priority of less critical network traffic classes to reduce congestion
Halted the datacenter automation software responsible for the maintenance event
Rescheduled the network control plane and its supporting infrastructure in each location
Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
Directed service traffic away from the affected regions
Moved the Cloud VPN control plane in three regions out of its fail-safe state
Action items
Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
Harden the cluster management software to reject such requests whatever their origin
Reconfigure the network control plane to reject maintenance requests of the type involved
Make the network control plane in each location persist its configuration
Let the network stay in fail static mode longer when the control plane is lost
Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
Extend continuous disaster recovery testing to this and similarly catastrophic failures
View case

Check your cluster for this

These 3 cases involve BGP routing and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

Graph

Cases3 cases

By root cause1

Software defect3 cases