Graph · Incident Library

Root cause
Component
Symptom
Contributing factor
Case

Cases39 cases

22 to 26 of 39

IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
Microsoft Azure2023
Symptoms
From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.
High latency Intermittent timeouts Intermittent connection failures Packet loss Network unreachable
Trigger
A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.
Root cause
Configuration change
Component
Backbone network (WAN)Interior routing (IGP)BGP routing
Contributing factor
No configuration tests Unverified assumption Excessive blast radius Harmful manual action
Detection
Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.
Remediation
Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
Action items
Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
Give on-call engineers real-time visibility of approved and unqualified activity on network devices
Run regular mandatory operational training and attestation of following all SOPs
Have the Change Advisory Board review all SOPs still pending qualification within 30 days
View case
IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were down
Wikimedia Foundation2023
Symptoms
On 10 January 2023 at 16:37 UTC Icinga reported "BGP status on cr3-eqsin is CRITICAL: BGP CRITICAL - AS1299/IPv6: Active - Telia, AS1299/IPv4: Active - Telia", and from 16:45 the hosts in Wikimedia's eqsin site went DOWN with "PING CRITICAL - Packet loss = 100%". Users normally directed to eqsin, mostly in the APAC region, could only read Wikipedia pages already cached in eqsin.
Network unreachable
Trigger
Planned maintenance by the transport provider took down eqsin's second transport link while the first was still down from a long fiber cut.
Root cause
Dependency failure
Component
Backbone network (WAN)BGP routing CDN and edge network
Contributing factor
Single point of failure Ignored early signal
Detection
Automated monitoring: Icinga BGP status alerts for cr3-eqsin at 16:37 and cr2-eqsin at 16:44, followed by a flood of host-down alerts and three pages at the same time.
Remediation
Depooled eqsin from DNS
Repooled eqsin
Action items
Create a backup GRE tunnel for eqsin (T327265)
Ensure the long-down transport link comes back up properly once it is fixed (T322529)
Automatically parse maintenance notifications and alert on conflicting maintenance (T230835)
Lessons
The site was depooled quickly, and "depool first, investigate later" was the right attitude. A flood of host-down alerts usually means a network problem. eqsin stayed too long on a single working transport link, SREs did not catch that the downtime of the two transport links overlapped, and the provider's planned maintenance was not recorded in a task or reminder.
View case
IL-0525 OpenStreetMap.org went offline for 7h 27m when its uplink switch lost power during a data-centre power feed maintenance
OpenStreetMap Foundation2022
Symptoms
From 20:18 UTC on 11 June 2022, OpenStreetMap.org was unavailable: all of its services hosted in Amsterdam, including www, the API, the wiki and planet, had no connectivity to the Internet. The outage began 18 minutes into Equinix's planned maintenance of power feed A at the Amsterdam data centre.
Network unreachable Service outage
Trigger
Equinix's planned 5-year maintenance of power feed A in Amsterdam, which started at 20:00 UTC on 11 June 2022.
Root cause
Hardware failure
Component
Network switch Data center power and cooling
Contributing factor
Single point of failure Limited debug access Monitoring blind spot
Remediation
Waited for the end of the Equinix power feed A maintenance window instead of failing over to Dublin or moving the uplink; connectivity returned at 03:45 UTC
Action items
Acknowledge ongoing outages faster
Add monitoring and alerting of switch PSU and FAN states
Replace the Cisco SG550X-48 switches and the Redundant Power System with Juniper EX4300 switches with redundant power supplies
Install out-of-band access equipment with dual power feeds in Amsterdam
Install redundant uplinks
Make site failover easier, less manual and lower risk
Introduce an on-call pager system for the operations SRE
Expand the SRE team
Continue and formalise fail-over tests for power, network, upstream link and service outages
View case
IL-0266 Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPs
Wikimedia Foundation2021
Symptoms
On 23 November 2021 at 09:39, two minutes after a change on the eqiad core routers, alerts fired because Icinga in eqiad could not reach the public text-lb address in eqsin (Singapore). For about 12 minutes eqiad could not reach hosts in other data centers, such as the cache PoPs, through their public IP addresses.
Network unreachable
Trigger
At about 09:37 a change on cr1-eqiad and cr2-eqiad, made as part of task T295672 to influence BGP route selection, removed a BGP setting that sets the BGP "MED" attribute to the OSPF cost of reaching the BGP route's next hop.
Root cause
Configuration change
Component
BGP routing
Contributing factor
Ineffective rollback
Detection
Icinga health checks against edge caching clusters alerted at 09:39; the first alert was Icinga in eqiad failing to reach the public text-lb address in eqsin.
Remediation
Reverted the configuration change on both eqiad routers
Forced a reset of the BGP session between the two eqiad routers
Action items
Updated T295672 with further detail on the incident and a proposal to adjust the BGP sessions to address the "second IXP port" issue and prevent this from happening again
View case
IL-0539 KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in Busan
KT2021
Symptoms
On 25 October 2021 from about 11:16, traffic to the DNS servers in KT's network rose sharply, and then KT's internet service failed across the whole country. KT recognised the internet outage at 11:20, and the outage lasted about 89 minutes until KT finished recovery at about 12:45.
Network unreachable Service outage
Trigger
During the replacement of an enterprise network router at KT's Busan office, worked on remotely from the Busan communication center, a worker entered a routing configuration script that lacked the 'exit' command that should have ended the IS-IS part of the configuration.
Root cause
Configuration change
Component
Interior routing (IGP)BGP routing
Contributing factor
Missed in review No configuration tests Change during peak hours Excessive blast radius
Detection
KT recognised the internet outage at 11:20, four minutes after DNS traffic began to surge at 11:16. The Ministry of Science and ICT saw the traffic increase in its own monitoring, and KT reported the outage to the ministry at 11:40.
Remediation
Isolated the router where the faulty routing configuration was entered
Logged in to each regional router, deleted the faulty processes and restarted them
Action items
Check the network management of major carriers, including their work procedures and the systems that stop technical errors from spreading
Consider having major carriers introduce simulation systems that diagnose errors from network work in advance
Have the NMS center and network operations center of each major carrier check technically that approved work plans are followed
Strengthen safeguards in routing work, such as limiting the number of route entries updated at once below a set level
Strengthen the monitoring of carriers' outages and develop technology, including AI-based preventive diagnosis, for network stability and resilience
Prepare structural measures for a stable network architecture and network survivability
Let carriers notify users of outages through SNS, text messages and other means, not only the website
Lessons
The investigation named management problems and technical problems. On the management side, work approved for the night was done during the day, partner company workers did the routing work without a work manager, and the work was done with the network still connected. On the technical side, the two pre-verification steps were people reading the script and missed the missing 'exit', there was no virtual testbed for trying the work on a disconnected network, and nothing stopped an error in one region from spreading to the whole country.
View case

Check your cluster for this

These 39 cases involve BGP routing and Network control plane (SDN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap