Incident Library
Applied filters ComponentCDN and edge networkSymptomHigh error rateClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases7 cases

4 to 7 of 7

  1. IL-0357 Up to 22% of Vercel static file requests failed when part of its global cache was lost during the AWS us-east-1 outage
    Vercel2025
    Symptoms

    On 20 October 2025 at 07:45 UTC, while Vercel was moving traffic away from its us-east-1 region (iad1) during the AWS us-east-1 outage, a cascading failure took out a percentage of Vercel's global caching infrastructure. Failures of static file serving peaked at about 22% of traffic, the largest impact of the whole event, until service was restored at 08:18 UTC.

    High error rate
    Trigger

    The AWS us-east-1 region (Vercel's iad1) failed on 20 October 2025, and Vercel began re-routing traffic and function invocations away from it; at 07:45 UTC a cascading failure followed in the global caching infrastructure.

    Root cause
    Cloud provider outage
    Component
    CDN and edge networkObject storage
    Contributing factor
    –
    Detection

    The first automated alerts, for errors across Vercel services, fired at 06:55 UTC at the start of the us-east-1 disruption, 50 minutes before the cache failure began; incidents were created automatically and on-call engineers were paged.

    Remediation
    • Re-routed end-user traffic away from iad1 into neighbouring data centers
    • Re-routed function invocations to configured backup or secondary regions
    • Restored static file serving at 08:18 UTC and mitigated the root cause of the cascading cache failure
    Action items
    • Write a detailed technical postmortem leading to systematic mitigation of the failure modes these disruptions exposed
    • Continue and intensify rehearsals of major regional disruptions
    View case
  2. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024
    Sentry2024
    Symptoms

    On 21 October 2024 Sentry could not ingest data in US SaaS from 9:42 AM to 10:06 AM PT, and the status page reported elevated errors for ingestion. The same failure had also happened on 4 and 16 October.

    High error rateService outageAutoscaler misbehaviour
    Trigger

    In each of the three incidents an ingestion region saw an increase in backlogged traffic; the trigger was different each time.

    Root cause
    Capacity exhaustion
    Component
    CDN and edge networkAutomated capacity manager
    Contributing factor
    Automation amplificationExcessive blast radius
    Action items
    • Create more, smaller edge regions to reduce the chance of cascading failures, adding one region each in South America, North America and Europe
    • Change the autoscaling rules to use stronger signals than CPU load
    • Scale down more slowly so that iowait-based backlogs do not trigger scale-down events
    • Improve the metrics the edge produces to refine the autoscaling rules and their sensitivity
    View case
  3. IL-0491 Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load
    Squarespace2019
    Symptoms

    At 12:10 AM ET on Tuesday, February 12, 2019, Squarespace's monitoring alerted the on-call engineer for the CDN origin fleet that the fleet had fallen below a safe number of healthy nodes. Requests to static.squarespace.com and static1.squarespace.com failed, so images, scripts and style sheets did not load for some sites, and the status page later reported elevated error rates serving media and stylesheets.

    High error rate
    Trigger

    At 10:30 PM ET on February 11, an engineer started a maintenance operation across the CDN origin fleet. The operation kept running in the background after the engineer believed it had been terminated.

    Root cause
    Operator error
    Component
    CDN and edge network
    Contributing factor
    Excessive blast radiusMonitoring blind spot
    Detection

    The monitoring system alerted the CDN origin fleet's on-call engineer at 12:10 AM ET when the fleet fell below a safe number of healthy nodes.

    Remediation
    • Restored the CDN origin fleet, partially by 2:15 AM ET and to normal behaviour at 2:20 AM ET
    Action items
    • Revise the incident-response playbooks with checks and actions for this fleet, starting with an assessment of customer impact and the immediate involvement of an Incident Commander
    • Improve external monitoring to assess customer impact of issues with the CDN partners and origin services more quickly
    • Revise operational policies and add safeguards for issuing maintenance commands, including limiting the share of the fleet a command can operate on
    • Reduce the need for direct manual maintenance operations
    • Create independent failure domains and route requests automatically around unhealthy parts of the system
    View case
  4. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018
    Symptoms

    From 12:17 to 12:49 PDT on 17 July 2018, Google Cloud HTTP(S) Load Balancers answered a share of requests with 502 return codes, and connections to TCP/SSL Proxy Load Balancers were reset. The share of failed requests moved between 33% and 87% during those 32 minutes, and automated monitoring alerted Google engineers at 12:19.

    High error rateConnection resetsHigh latency
    Trigger

    A configuration change in the production environment intermittently set off a bug in a new feature of the second-layer Google Front Ends.

    Root cause
    Software defect
    Component
    Cloud load balancerCDN and edge network
    Contributing factor
    Gap in automated testsExcessive blast radius
    Detection

    Automated monitoring alerted Google's engineering team at 12:19 PDT, two minutes after the 502s began.

    Remediation
    • Reverted the configuration change at 12:44 PDT, which stopped the GFE restarts
    Action items
    • Add safeguards that disable features not yet in service
    • Harden the GFE testing stack against latent bugs in production binaries that make a task restart
    • Isolate the shards of GFE pools further from each other to reduce the scope of failures
    • Build a consolidated dashboard of all configuration changes for GFE pools
    View case
Check your cluster for this

These 7 cases involve CDN and edge network and Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Dependency failure1 case

Component
CDN and edge networkObject storage
Contributing factor
Signal buried in noiseOverlapping failuresPartial rollout
Organisation
Python Software Foundation
  1. IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installs
    Python Software Foundation2026

Cloud provider outage2 cases

Component
CDN and edge network2Cloud load balancer1Ingress controller1Object storage1Web application firewall rules1
Contributing factor
Single point of failure1
Organisation
ANDPAD1Vercel1
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
  2. IL-0357 Up to 22% of Vercel static file requests failed when part of its global cache was lost during the AWS us-east-1 outage
    Vercel2025

Configuration change1 case

Component
CDN and edge networkDatabase
Contributing factor
Unverified assumptionNo configuration testsNo staged rolloutOverlapping failuresPartial rolloutRetry storm
Organisation
Cloudflare
  1. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file
    Cloudflare2025

Capacity exhaustion1 case

Component
Automated capacity managerCDN and edge network
Contributing factor
Automation amplificationExcessive blast radius
Organisation
Sentry
  1. IL-0555 Sentry's ingestion edge regions backlogged one after another and rejected traffic in three incidents in October 2024
    Sentry2024

Operator error1 case

Component
CDN and edge network
Contributing factor
Excessive blast radiusMonitoring blind spot
Organisation
Squarespace
  1. IL-0491 Squarespace's CDN origin fleet fell below its healthy node threshold, and images, scripts and stylesheets failed to load
    Squarespace2019

Software defect1 case

Component
CDN and edge networkCloud load balancer
Contributing factor
Excessive blast radiusGap in automated tests
Organisation
Google Cloud
  1. IL-0338 Google Cloud HTTP(S) and TCP/SSL Proxy load balancers returned 502s or reset connections on 33% to 87% of requests
    Google Cloud2018