Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018
    Symptoms

    From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

    High error rateConnection resetsHigh latency
    Trigger

    A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

    Root cause
    Software defect
    Component
    Cloud load balancerCDN and edge network
    Contributing factor
    Gap in automated testsExcessive blast radius
    Detection

    Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

    Remediation
    • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts
    Action items
    • Add safeguards that disable features not yet in service
    • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
    • Isolate the shards of GFE pools further from each other to reduce the scope of failures
    • Build a consolidated dashboard of all configuration changes for GFE pools
    View case
Check your cluster for this

These 6 cases involve CDN and edge network and BGP routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Software defect2 cases

Component
CDN and edge network2Cloud load balancer1Authoritative DNS servers and records1
Contributing factor
Excessive blast radius2Gap in automated tests2Change with a delayed effect1Ineffective rollback1
Organisation
Google Cloud1Microsoft Azure1
  1. IL-0226 Azure Front Door edge servers crashed worldwide on incompatible configuration metadata, causing timeouts and DNS errors
    Microsoft Azure2025
  2. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018

Capacity exhaustion2 cases

Component
CDN and edge network2BGP routing1Automated capacity manager1Transit gateway or network interconnect1
Contributing factor
Excessive blast radius2Automation amplification1Harmful manual action1No headroom1
Organisation
Cloudflare1Sentry1
  1. IL-0341 Traffic between Cloudflare and AWS us-east-1 had high latency, packet loss and origin failures for almost four hours
    Cloudflare2025
  2. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024
    Sentry2024

Configuration change1 case

Component
CDN and edge networkData center power and coolingDDoS protection serviceNetwork control plane (SDN)
Contributing factor
Excessive blast radiusMonitoring blind spotOverlapping failures
Organisation
Microsoft Azure
  1. IL-0339 Azure Front Door and Azure CDN users saw intermittent connection errors, timeouts and latency spikes for hours
    Microsoft Azure2024

Operator error1 case

Component
CDN and edge network
Contributing factor
Excessive blast radiusMonitoring blind spot
Organisation
Squarespace
  1. IL-0491 Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load
    Squarespace2019