Incident Library
Applied filters Root causeDependency failureComponentCDN and edge networkSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task
    Canva2024
    Symptoms

    On 12 November 2024, after a new version of Canva's editor was deployed at 8:47 AM UTC, the editor's object panel stayed in a perpetual loading state for many users, particularly in Asia. Then from 9:08 AM UTC to about 10:00 AM UTC canva.com was unavailable and all requests to it failed.

    Service outageOOMKilledHigh latencyPacket loss
    Trigger

    At 8:47 AM UTC a new version of the Canva editor page was deployed, and client devices began fetching its new static assets through Cloudflare while the network path between Cloudflare's Singapore and Ashburn locations was losing packets.

    Root cause
    Dependency failure
    Component
    CDN and edge networkAPI gatewayAmazon ECS
    Contributing factor
    Overlapping failuresMonitoring blind spotRetry stormAutomation amplificationUnpatched known bug
    Remediation
    • Added a temporary Cloudflare firewall rule at 9:29 AM UTC that blocked all traffic at the CDN so new API Gateway tasks could start
    • Redirected canva.com to the status page
    • Restored traffic gradually, starting with Australian users under strict rate limits
    Action items
    • Write a runbook for rerouting, blocking and progressively restoring traffic
    • Show users a more informative error page when canva.com is unavailable
    • Raise the baseline number of API Gateway tasks and the memory of each task
    • Add load shedding rules to the API Gateway for similar traffic patterns
    • Load test the API Gateway regularly
    • Deploy the patch for the thread-locking bug in the telemetry library
    • Add multithreaded tests for the contended code path to the telemetry library microbenchmark suite
    • Use page load completion events as a second canary indicator for page asset releases
    • Experiment with adding another rollout stage to the canary to give more time to detect issues
    • Add timeouts so user requests do not wait excessively long for assets
    • Work with Cloudflare to understand the system interactions behind the incident
    View case
  2. IL-0398 GOV.UK showed "Error 503 Service Unavailable" pages for under an hour during a global outage of its primary CDN
    Government Digital Service2021
    Symptoms

    On 8 June 2021, for less than an hour, people visiting GOV.UK could not reach its information and services and got error pages with the heading "Error 503 Service Unavailable" instead. The outage was in the primary content delivery network (CDN) that GOV.UK uses, and many other websites on the same CDN around the world were down at the same time.

    Service outage
    Root cause
    Dependency failure
    Component
    CDN and edge network
    Contributing factor
    –
    Detection

    The incident was detected 4 minutes after it began, and an incident lead and a communications lead then started the formal incident process.

    Remediation
    • Waited 15 minutes to judge the primary CDN outage, then decided to fail over to the secondary CDN
    • Deployed the prepared DNS record changes, and traffic began to route to the secondary CDN within 30 minutes
    • Switched back to the primary CDN after it recovered, and monitored for an hour before declaring the incident resolved
    Action items
    • Hold a formal incident review of what happened and how the team responded
    • Consider how to improve the speed and transparency of communications
    • Consider how planned changes to platform technology and processes can make recovery from future incidents faster
    Lessons

    The CDN sits at the edge of the network, so when it breaks the whole website goes down, and the team keeps an always-on secondary CDN and a prepared DNS change for that case. Failing over too early can make an incident worse, because the secondary CDN gives a degraded service and primary CDN incidents are often resolved quickly, so the plan waits 15 minutes before failing over.

    View case
Check your cluster for this

These 2 cases involve CDN and edge network and API gateway. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure2 cases

Component
CDN and edge network2API gateway1Amazon ECS1
Contributing factor
Automation amplification1Unpatched known bug1Monitoring blind spot1Overlapping failures1Retry storm1
Organisation
Canva1Government Digital Service1
  1. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task
    Canva2024
  2. IL-0398 GOV.UK showed "Error 503 Service Unavailable" pages for under an hour during a global outage of its primary CDN
    Government Digital Service2021