Incident Library

Applied filters Root causeConfiguration changeSymptomPacket lossClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases5 cases

1 to 5 of 5

IL-0394 A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backbone
Cloudflare2026
Symptoms
On 22 January 2026 from 20:25 UTC, a Cloudflare edge router in Miami sent unintended BGP advertisements of IPv6 prefixes to its transit providers and peers. External networks' traffic was funnelled into the Miami data center, the backbone between Miami and Atlanta became congested, and some Cloudflare customer traffic saw elevated loss and higher latency.
Packet loss High latency
Trigger
At 20:25 UTC Cloudflare's policy automation pushed a change, merged at 19:52 UTC, that removed the BGP announcements from Miami for a data center in Bogotá, Colombia, by deleting the prefix-list 6-BOG04-SITE-LOCAL from several export policy terms.
Root cause
Configuration change
Component
BGP routing Backbone network (WAN)
Contributing factor
No configuration tests
Detection
The network team began investigating unintended route advertisements from Miami at 20:40 UTC, and an incident was raised at 20:44 UTC.
Remediation
A network operator reverted the bad configuration change on the Miami router by hand
Paused the automation for the router so it could not run again
Reverted the triggering change in the network automation code repository
Unpaused the automation on the Miami router after confirming it ran without the routing policy bug
Action items
Patch the failure in the routing policy automation that caused the leak
Add BGP community-based safeguards that reject routes received from providers and peers on external export policies
Add automatic routing policy evaluation to the CI/CD pipelines that looks for empty or erroneous policy terms
Detect problems in network configurations and the effects of automated changes earlier
Validate routing equipment vendors' implementations of RFC9234 before rolling out BGP roles and the Only-to-Customer Attribute
Encourage long-term adoption of RPKI Autonomous System Provider Authorization (ASPA)
Lessons
Removing only the prefix lists from export policy terms left terms that were too permissive, because a match on route-type internal accepts IBGP routes as well. Cloudflare states that route leaks are easy to cause by accident and that RFC9234 is the only way, independent of routing policy, to prevent route leaks caused at the local AS.
View case
IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours
Microsoft Azure2024
Symptoms
From 11:45 UTC on 30 July 2024, a subset of customers got intermittent connection errors, timeouts and latency spikes when connecting to Microsoft services that use Azure Front Door (AFD) and Azure Content Delivery Network (CDN). The Azure Portal team saw the first service degradation at 11:47 UTC.
Intermittent connection failures Intermittent timeouts High latency Packet loss
Trigger
Around 11:45 UTC the Azure network DDoS protection service was disengaging after a routine DDoS mitigation and returning traffic to its default routing to AFD, but the routes at one site in Europe could not be updated because the network control plane could not reach that site during a local power outage.
Root cause
Configuration change
Component
CDN and edge network DDoS protection service Network control plane (SDN)Data center power and cooling
Contributing factor
Overlapping failures Excessive blast radius Monitoring blind spot
Detection
The Azure Portal team detected service degradation at 11:47 UTC and began to investigate. At 12:10 UTC network monitoring correlated the Portal incident to a network issue at one site in Europe, and networking engineers joined.
Remediation
Rerouted traffic under the standard congestion playbook, which removed most of the impact by 13:58 UTC
Ordered the network control plane to reissue its commands to the misbehaving device, which could not be reached
Removed the device that kept attracting traffic from the network and scanned the network for other issues
Action items
Add the missing configuration on network devices so a DDoS mitigation issue in one geography cannot spread to another
Improve validation and monitoring in the Azure network to detect invalid configurations
Improve monitoring for when the DDoS protection service is unreachable from the control plane but still serving traffic
View case
IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
Microsoft Azure2023
Symptoms
From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.
High latency Intermittent timeouts Intermittent connection failures Packet loss Network unreachable
Trigger
A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.
Root cause
Configuration change
Component
Backbone network (WAN)Interior routing (IGP)BGP routing
Contributing factor
No configuration tests Unverified assumption Excessive blast radius Harmful manual action
Detection
Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.
Remediation
Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
Action items
Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
Give on-call engineers real-time visibility of approved and unqualified activity on network devices
Run regular mandatory operational training and attestation of following all SOPs
Have the Change Advisory Board review all SOPs still pending qualification within 30 days
View case
IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
Datadog2022
Symptoms
From September 2021, each rollout of an update to Datadog's metrics query service, which serves real-time and historical metric data to the web frontend and to monitor evaluation, came with a rise in errors. Clients retried failed queries, so few errors reached users, but the retries slowed dashboards and monitors. The service's logs showed DNS errors stopping it from connecting to the data stores it depends on.
High error rate DNS resolution failures High latency OOMKilled Packet loss Stale endpoint routing
Trigger
Rollouts of the metrics query service, which delete the old pods. The client behaviour behind the failures came from an August 2021 rollout of the metrics query clients that applied round_robin as the new default gRPC load balancing policy.
Root cause
Configuration change
Component
conntrack table Cilium Linux kernel CoreDNS Client-side DNS resolver and cache external-dns HTTP and RPC client library
Contributing factor
Change with a delayed effect Retry storm Unknown limit Monitoring blind spot
Detection
Errors rose during rollouts of the metrics query service, and the service's logs showed DNS errors when connecting to its data stores.
Remediation
Raised the memory of the node-local-dns pods from 64 MB to 256 MB, which stopped the OOM kills but not the DNS errors
Tested larger instance types and confirmed that scaling up the service's infrastructure resolved the errors
Removed the custom gRPC reconnect parameters from the clients to restore the defaults
Action items
Contribute a change to Cilium that adds an unreachable route for a deleted pod's IP so clients get an ICMP error
Lessons
The errors looked like DNS but were not. ENA metrics and VPC Flow Logs gave the visibility into VPC connection tracking that host metrics did not, and Cilium, Kubernetes and AWS networking interact in ways that can make reverse path filtering drop packets it should allow. DNS propagation time depends on many settings. The causes were changes Datadog had made itself, to work around a dependency bug, to resolve earlier incidents and to override default reconnect parameters, which looked safe at the time and showed their effects much later, when the link was hard to see.
View case
IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
SEEK2021
Symptoms
In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.
High error rate Packet loss Intermittent timeouts
Trigger
In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.
Root cause
Configuration change
Component
Transit gateway or network interconnect
Contributing factor
Unknown limit Unreviewed defaults Monitoring blind spot
Detection
The team noticed soon after the routing change that search on seek.com.au was not working.
Remediation
Rolled back the migration of the search traffic's peering connection to the transit gateway
Action items
Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up
Lessons
A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.
View case

Check your cluster for this

These 5 cases involve Backbone network (WAN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change5 cases

Component: Backbone network (WAN)2 BGP routing2 CDN and edge network1 Cilium1 conntrack table1 CoreDNS1 Data center power and cooling1 DDoS protection service1 Client-side DNS resolver and cache1 external-dns1 HTTP and RPC client library1 Interior routing (IGP)1 Transit gateway or network interconnect1 Linux kernel1 Network control plane (SDN)1
Contributing factor: Monitoring blind spot3 Excessive blast radius2 Unknown limit2 No configuration tests2 Unverified assumption1 Unreviewed defaults1 Change with a delayed effect1 Harmful manual action1 Overlapping failures1 Retry storm1
Organisation: Microsoft Azure2 Cloudflare1 Datadog1 SEEK1