Incident Library

Applied filters SymptomNetwork unreachableClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases39 cases

27 to 31 of 39

IL-0228 IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPF
OVHcloud2021
Symptoms
At 09:20 CET on 13 October 2021, during a planned change on a router, IPv4 traffic across OVHcloud's whole network stopped being routed, while all IPv6 traffic stayed reachable. The team detected a problem with the router's performance at 09:21 CET and escalated at once.
Network unreachable Service outage
Trigger
A planned change to a router in the Vint Hill datacentre in the United States, meant to strengthen anti-DDoS protection, included a command to regulate the redistribution of BGP into OSPF, and the router did not interpret that command correctly.
Root cause
Configuration change
Component
Interior routing (IGP)BGP routing Backbone network (WAN)
Contributing factor
Ineffective rollback Excessive blast radius
Detection
At 09:21 CET the team detected a problem with the router's performance and started the escalation process.
Remediation
Disconnected the optical equipment to isolate the router's connectivity
Powered off the faulty router
Stabilised connectivity to restore the remaining services
Action items
Re-evaluate the change procedure for this type of equipment, which applies the command line natively
Strengthen the related change process
View case
IL-0221 A backbone maintenance command disconnected all Facebook data centers, and its DNS servers withdrew their BGP routes
Meta2021
Symptoms
On 4 October 2021 Facebook's platforms went down worldwide: every connection in Facebook's global backbone network was lost, and the data centers were cut off from each other and from the internet. The rest of the internet could no longer find Facebook's servers, because Facebook's authoritative DNS servers had become unreachable.
Service outage DNS resolution failures Network unreachable
Trigger
During routine maintenance of the backbone, a command was issued to assess the availability of global backbone capacity, and it unintentionally took down all connections in the backbone network.
Root cause
Operator error
Component
Backbone network (WAN)BGP routing Authoritative DNS servers and records
Contributing factor
Automation amplification Circular dependency Limited debug access
Remediation
Sent engineers onsite to the data centers to debug the issue and restart systems
Brought the backbone network back online across the data center regions
Brought services back gradually, managing the rising load to avoid crashes from a traffic surge
Action items
Find ways to simulate the global backbone being taken offline in storm drills
Strengthen testing, drills and overall resilience
Lessons
The hardening that protects Facebook's data centers against unauthorized access slowed recovery from an outage caused by Facebook's own error, and the author judges that tradeoff worth keeping. Storm drills prepared the teams to bring load back safely, although no drill had yet simulated the global backbone going offline.
View case
IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
Google Cloud2019
Symptoms
On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.
Packet loss Network unreachable High latency High error rate Monitoring outage
Trigger
At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.
Root cause
Software defect
Component
Network control plane (SDN)BGP routing Backbone network (WAN)
Contributing factor
Excessive blast radius Circular dependency Limited debug access Ineffective rollback
Detection
Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.
Remediation
Lowered the priority of less critical network traffic classes to reduce congestion
Halted the datacenter automation software responsible for the maintenance event
Rescheduled the network control plane and its supporting infrastructure in each location
Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
Directed service traffic away from the affected regions
Moved the Cloud VPN control plane in three regions out of its fail-safe state
Action items
Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
Harden the cluster management software to reject such requests whatever their origin
Reconfigure the network control plane to reject maintenance requests of the type involved
Make the network control plane in each location persist its configuration
Let the network stay in fail static mode longer when the control plane is lost
Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
Extend continuous disaster recovery testing to this and similarly catastrophic failures
View case
IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic
Packet2019
Symptoms
At about 08:49 UTC on 2 June 2019, Packet's internal network monitoring alerted on serious reachability and performance problems in several of Packet's global datacenter locations. The first investigation found routing loops and traffic being blackholed to specific internal and Internet destinations.
Network unreachable Packet loss High latency
Trigger
A trans-continental US backbone link in the network of Packet's upstream provider Zayo had a hardware failure and blackholed IP traffic routed over it.
Root cause
Dependency failure
Component
Backbone network (WAN)BGP routing Network control plane (SDN)
Contributing factor
–
Detection
Packet's internal network monitoring alerted at 08:49 UTC on reachability issues in several global datacenter locations, and engineers confirmed them at 08:52.
Remediation
Migrated traffic to an alternative network and removed Zayo from the provider mix
Applied route filters to return service to normal
Optimised specific routing policies to improve latency across the backbone with the new traffic mix
Action items
Expand internal connectivity monitoring and alerting of on-call operations teams
Expand network automation and routing so traffic can be balanced across all available backbone paths and an operator can trigger automated re-routes across all Packet datacenters in a major provider incident
Make Packet's provisioning systems less dependent on third party network providers and inter-datacenter connectivity
View case
IL-0245 Acquia Cloud Enterprise sites in US-East went down when their web servers could no longer reach the Memcache servers
Acquia2019
Symptoms
On 7 February 2019 from 22:17 to 23:25 UTC, some Acquia Cloud Enterprise sites in US-East became unstable or went down for multiple customers. The web servers could not reach the Memcache servers, PHP processes on the web servers were blocked, and the sites became unavailable.
Service outage Network unreachable
Trigger
While setting up a temporary file system to store diagnostic data, a rule was deleted by mistake from a VPC security group.
Root cause
Operator error
Component
VPC route tables and firewall rules In-memory cache
Contributing factor
Ignored early signal
Detection
Acquia's configuration drift alerting fired, but the alert was not seen in time to shorten the outage.
Remediation
Restored the missing rule in the security group by hand
Action items
Correct the internal workflow with all involved teams to prevent similar mistakes
Alert internal teams faster about configuration drift in VPC security groups
View case

Check your cluster for this

These 39 cases involve BGP routing and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap