Incident Library
Applied filters SymptomPacket lossFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

6 to 7 of 7

  1. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023
    Symptoms

    From 07:08 UTC on 25 January 2023, customers saw long network latency and timeouts when connecting to resources in Azure regions and to other Microsoft services such as Microsoft 365 and Power Platform. Microsoft's monitoring detected DNS and WAN issues from 07:11 UTC.

    High latencyIntermittent timeoutsIntermittent connection failuresPacket lossNetwork unreachable
    Trigger

    A network engineer adding capacity to the global WAN in Madrid ran a step of a recently changed standard operating procedure that purged the IGP database; on the router manufacturer involved, that command runs on all IGP-joined routers instead of only the local one.

    Root cause
    Configuration change
    Component
    Backbone network (WAN)Interior routing (IGP)BGP routing
    Contributing factor
    No configuration testsUnverified assumptionExcessive blast radiusHarmful manual action
    Detection

    Monitoring detected DNS and WAN issues from 07:11 UTC and alerted the on-call engineers within 5 minutes of the command being run. The team reviewed all recent changes and identified the command by 08:20 UTC. The engineer making the change was not informed of the alerts because of the unqualified changes to the SOP.

    Remediation
    • Restarted by hand the paused WAN health and traffic engineering systems, which ended the remaining packet loss
    Action items
    • Audit and block similar widespread-impact commands across all three vendors for all WAN router roles
    • Give on-call engineers real-time visibility of approved and unqualified activity on network devices
    • Run regular mandatory operational training and attestation of following all SOPs
    • Have the Change Advisory Board review all SOPs still pending qualification within 30 days
    View case
  2. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019
    Symptoms

    On Sunday 2 June 2019, with user impact beginning between 11:47 and 11:49 US/Pacific, Google Cloud projects in several US regions saw elevated packet loss from network congestion, and some regions became unreachable. Customers saw higher latency, intermittent errors and lost connectivity to instances in us-central1, us-east1, us-east4, us-west2, northamerica-northeast1 and southamerica-east1, while us-west1, Europe and Asia had no regional congestion.

    Packet lossNetwork unreachableHigh latencyHigh error rateMonitoring outage
    Trigger

    At 11:45 US/Pacific a relatively rare type of datacenter maintenance event started in one physical location, and the automation that runs it descheduled logical clusters that also held network control plane jobs in other physical locations.

    Root cause
    Software defect
    Component
    Network control plane (SDN)BGP routingBackbone network (WAN)
    Contributing factor
    Excessive blast radiusCircular dependencyLimited debug accessIneffective rollback
    Detection

    Google engineers were alerted to the failure two minutes after it began and started the incident process used for the most serious production incidents. The first customer notice appeared on the status page at 12:53 US/Pacific.

    Remediation
    • Lowered the priority of less critical network traffic classes to reduce congestion
    • Halted the datacenter automation software responsible for the maintenance event
    • Rescheduled the network control plane and its supporting infrastructure in each location
    • Rebuilt and redistributed the lost network control plane configuration, rolling it out from 14:03
    • Directed service traffic away from the affected regions
    • Moved the Cloud VPN control plane in three regions out of its fail-safe state
    Action items
    • Keep the maintenance automation off until safeguards stop it descheduling jobs in several physical locations at once
    • Harden the cluster management software to reject such requests whatever their origin
    • Reconfigure the network control plane to reject maintenance requests of the type involved
    • Make the network control plane in each location persist its configuration
    • Let the network stay in fail static mode longer when the control plane is lost
    • Review, update and test emergency response tooling and procedures, including customer communication tooling, for such network failures
    • Extend continuous disaster recovery testing to this and similarly catastrophic failures
    View case
Check your cluster for this

These 7 cases involve Network control plane (SDN) and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Capacity exhaustion1 case

Component
BGP routingCDN and edge networkTransit gateway or network interconnect
Contributing factor
Excessive blast radiusHarmful manual actionNo headroom
Organisation
Cloudflare
  1. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025

Software defect3 cases

Component
Network control plane (SDN)3BGP routing2Backbone network (WAN)1Cloud load balancer1Database1Managed Kubernetes control plane and provider API1NAT gateway1Network switch1
Contributing factor
Excessive blast radius3Automation amplification1Circular dependency1Change with a delayed effect1Ignored early signal1Limited debug access1No safeguard on destructive change1Ineffective rollback1
Organisation
Google Cloud2Scaleway1
  1. IL-0347 About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuck
    Scaleway2024
  2. IL-0529 New Google Cloud VMs and other instances came up without network connectivity in all zones after VPC controllers stopped
    Google Cloud2024
  3. IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduled
    Google Cloud2019

Configuration change2 cases

Component
Backbone network (WAN)1BGP routing1CDN and edge network1Data center power and cooling1DDoS protection service1Interior routing (IGP)1Network control plane (SDN)1
Contributing factor
Excessive blast radius2Unverified assumption1Harmful manual action1Monitoring blind spot1No configuration tests1Overlapping failures1
Organisation
Microsoft Azure2
  1. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours
    Microsoft Azure2024
  2. IL-0225 An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet loss
    Microsoft Azure2023

Faulty deployment1 case

Component
Cloud load balancerNetwork control plane (SDN)
Contributing factor
Excessive blast radiusNo alert on limit
Organisation
Google Cloud
  1. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023