Graph
Cases6 cases
1 to 5 of 6
IL-0400Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default routePSBA2026- Symptoms
On 25 March 2026 from about 16:40, public sector organisations on the PSBA network across Wales lost connectivity at their sites. The Service Desk saw about 400 alerts at 16:58, when a Major Incident was declared, about 1,500 at 17:05 and about 5,000 across Wales at 17:08, while councils, a fire and rescue service and Natural Resources Wales phoned in to report sites down.
- Trigger
At 16:30 a route map change was applied, outside formal change control, on a PSBA core Provider Edge (PE) router to leak MPLS VPN routes, so that legacy Capita-operated UPS devices in a /16 subnet could be managed by BT/PSBA. The change was removed at 16:40.
- Root cause
- Component
- Contributing factor
- Detection
At 16:57 Mid Wales Fire & Rescue Service phoned in a loss of connectivity at its Presteigne site, and at 16:58 a case was opened in Service Now for a group of about 400 alerts and a Major Incident was declared.
- Remediation
- Removed the route map change from the PE router at 16:40
- Changed QoS on the BGP to bring back the management links, after which alarms cleared
- Cleared routes and reset inbound BGP advertisements, which did not stop the default route being advertised
- Corrected the configuration issue affecting the internet learned route at 01:02
- Performed a controlled reload of the affected router at 01:10, which restored customer services and management connectivity
- Action items
- Reinforce change control requirements with an internal briefing on formal change submission, the risks of unplanned changes and roles during planned and emergency changes
- Create a dedicated Microsoft Teams channel as the single real-time communication point for all live P1 and Major Incidents
- Explore disaster recovery and business continuity test exercises for major or P1 incidents
- During loss of management access, have all technical teams assess customer impact from what reporting organisations provide, captured and shared in real time by the Service Desk and Incident Management
- Investigate a fixed link between Capital Quarter and the PSBA management racks in Vantage Data Centre at Newport, with dedicated management terminals and two-factor authentication for access outside the secure management platform
- Improve initial triage with stronger probing questions when incidents are reported and keep reassessing and communicating impact throughout the incident
- Review the P1 and Major Incident process so escalation criteria and timeframes are clearly defined, and train the Service Desk to recognise and escalate them early
- Review the PSBA estate for legacy configuration, set processes and timeframes for removing it and add controls so it is not introduced again
- Review email and SMS distribution lists and processes and reinforce Service Desk training on customer communications during P1 and Major Incidents
- Lessons
The report names eight lessons: a change was made outside PSBA change control; internal communication during the Major Incident was slow and fragmented; the organisation was not ready for a network-wide incident; losing management access stopped the resolver groups from seeing the network state and customer impact; the resolver groups did not understand customer impact in enough detail, which delayed finding the root cause; escalation to PSBA management was slow; legacy configuration left in the network contributed; and email and SMS communications did not reach some customers in time.
IL-0394A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backboneCloudflare2026- Symptoms
On 22 January 2026 from 20:25 UTC, a Cloudflare edge router in Miami sent unintended BGP advertisements of IPv6 prefixes to its transit providers and peers. External networks' traffic was funnelled into the Miami data center, the backbone between Miami and Atlanta became congested, and some Cloudflare customer traffic saw elevated loss and higher latency.
- Trigger
At 20:25 UTC Cloudflare's policy automation pushed a change, merged at 19:52 UTC, that removed the BGP announcements from Miami for a data center in Bogotá, Colombia, by deleting the prefix-list 6-BOG04-SITE-LOCAL from several export policy terms.
- Root cause
- Component
- Contributing factor
- Detection
The network team began investigating unintended route advertisements from Miami at 20:40 UTC, and an incident was raised at 20:44 UTC.
- Remediation
- A network operator reverted the bad configuration change on the Miami router by hand
- Paused the automation for the router so it could not run again
- Reverted the triggering change in the network automation code repository
- Unpaused the automation on the Miami router after confirming it ran without the routing policy bug
- Action items
- Patch the failure in the routing policy automation that caused the leak
- Add BGP community-based safeguards that reject routes received from providers and peers on external export policies
- Add automatic routing policy evaluation to the CI/CD pipelines that looks for empty or erroneous policy terms
- Detect problems in network configurations and the effects of automated changes earlier
- Validate routing equipment vendors' implementations of RFC9234 before rolling out BGP roles and the Only-to-Customer Attribute
- Encourage long-term adoption of RPKI Autonomous System Provider Authorization (ASPA)
- Lessons
Removing only the prefix lists from export policy terms left terms that were too permissive, because a match on route-type internal accepts IBGP routes as well. Cloudflare states that route leaks are easy to cause by accident and that RFC9234 is the only way, independent of routing policy, to prevent route leaks caused at the local AS.
IL-0225An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet lossMicrosoft Azure2023- Symptoms
From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.
- Trigger
A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.
- Root cause
- Component
- Contributing factor
- Detection
Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.
- Remediation
- Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
- Action items
- Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
- Give on-call engineers real-time visibility of approved and unqualified activity on network devices
- Run regular mandatory operational training and attestation of following all SOPs
- Have the Change Advisory Board review all SOPs still pending qualification within 30 days
- Symptoms
On 23 November 2021 at 09:39, two minutes after a change on the eqiad core routers, alerts fired because Icinga in eqiad could not reach the public text-lb address in eqsin (Singapore). For about 12 minutes eqiad could not reach hosts in other data centers, such as the cache PoPs, through their public IP addresses.
- Trigger
At about 09:37 a change on cr1-eqiad and cr2-eqiad, made as part of task T295672 to influence BGP route selection, removed a BGP setting that sets the BGP "MED" attribute to the OSPF cost of reaching the BGP route's next hop.
- Root cause
- Component
- Contributing factor
- Detection
Icinga health checks against edge caching clusters alerted at 09:39; the first alert was Icinga in eqiad failing to reach the public text-lb address in eqsin.
- Remediation
- Reverted the configuration change on both eqiad routers
- Forced a reset of the BGP session between the two eqiad routers
- Action items
- Updated T295672 with further detail on the incident and a proposal to adjust the BGP sessions to address the "second IXP port" issue and prevent this from happening again
IL-0539KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in BusanKT2021- Symptoms
On 25 October 2021 from about 11:16, traffic to the DNS servers in KT's network rose sharply, and then KT's internet service failed across the whole country. KT recognised the internet outage at 11:20, and the outage lasted about 89 minutes until KT finished recovery at about 12:45.
- Trigger
During the replacement of an enterprise network router at KT's Busan office, worked on remotely from the Busan communication center, a worker entered a routing configuration script that lacked the 'exit' command that should have ended the IS-IS part of the configuration.
- Root cause
- Component
- Contributing factor
- Detection
KT recognised the internet outage at 11:20, four minutes after DNS traffic began to surge at 11:16. The Ministry of Science and ICT saw the traffic increase in its own monitoring, and KT reported the outage to the ministry at 11:40.
- Remediation
- Isolated the router where the faulty routing configuration was entered
- Logged in to each regional router, deleted the faulty processes and restarted them
- Action items
- Check the network management of major carriers, including their work procedures and the systems that stop technical errors from spreading
- Consider having major carriers introduce simulation systems that diagnose errors from network work in advance
- Have the NMS center and network operations center of each major carrier check technically that approved work plans are followed
- Strengthen safeguards in routing work, such as limiting the number of route entries updated at once below a set level
- Strengthen the monitoring of carriers' outages and develop technology, including AI-based preventive diagnosis, for network stability and resilience
- Prepare structural measures for a stable network architecture and network survivability
- Let carriers notify users of outages through SNS, text messages and other means, not only the website
- Lessons
The investigation named management problems and technical problems. On the management side, work approved for the night was done during the day, partner company workers did the routing work without a work manager, and the work was done with the network still connected. On the technical side, the two pre-verification steps were people reading the script and missed the missing 'exit', there was no virtual testbed for trying the work on a disconnected network, and nothing stopped an error in one region from spreading to the whole country.
These 6 cases involve BGP routing and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.
By root cause1
Configuration change6 cases
- Contributing factor
IL-0400Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default routePSBA2026IL-0394A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backboneCloudflare2026IL-0225An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet lossMicrosoft Azure2023IL-0266Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPsIL-0539KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in BusanKT2021IL-0228IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPFOVHcloud2021