Incident Library
Applied filters Root causeDependency failureFactorRetry stormClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task
    Canva2024
    Symptoms

    On 12 November 2024, after a new version of Canva's editor was deployed at 8:47 AM UTC, the editor's object panel stayed in a perpetual loading state for many users, particularly in Asia. Then from 9:08 AM UTC to about 10:00 AM UTC canva.com was unavailable and all requests to it failed.

    Service outageOOMKilledHigh latencyPacket loss
    Trigger

    At 8:47 AM UTC a new version of the Canva editor page was deployed, and client devices began fetching its new static assets through Cloudflare while the network path between Cloudflare's Singapore and Ashburn locations was losing packets.

    Root cause
    Dependency failure
    Component
    CDN and edge networkAPI gatewayAmazon ECS
    Contributing factor
    Overlapping failuresMonitoring blind spotRetry stormAutomation amplificationUnpatched known bug
    Remediation
    • Added a temporary Cloudflare firewall rule at 9:29 AM UTC that blocked all traffic at the CDN so new API Gateway tasks could start
    • Redirected canva.com to the status page
    • Restored traffic gradually, starting with Australian users under strict rate limits
    Action items
    • Write a runbook for rerouting, blocking and progressively restoring traffic
    • Show users a more informative error page when canva.com is unavailable
    • Raise the baseline number of API Gateway tasks and the memory of each task
    • Add load shedding rules to the API Gateway for similar traffic patterns
    • Load test the API Gateway regularly
    • Deploy the patch for the thread-locking bug in the telemetry library
    • Add multithreaded tests for the contended code path to the telemetry library microbenchmark suite
    • Use page load completion events as a second canary indicator for page asset releases
    • Experiment with adding another rollout stage to the canary to give more time to detect issues
    • Add timeouts so user requests do not wait excessively long for assets
    • Work with Cloudflare to understand the system interactions behind the incident
    View case
  2. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020
    Symptoms

    At 11:59AM CDT on Friday 28 August 2020, Basecamp's Ops team received a flood of alerts from PagerDuty, Nagios and Prometheus. Monitoring showed that the newer of the two metro links between the data centers in Ashburn, VA and Chicago, IL had been lost for about 20-30 seconds, and BC3 kept responding slowly after the link came back.

    Link flappingHigh latencyReplication lag
    Trigger

    Both redundant metro links between Basecamp's Ashburn and Chicago data centers dropped for short periods; monitoring showed the newer link down for about 20-30 seconds.

    Root cause
    Dependency failure
    Component
    Backbone network (WAN)BGP routingIn-memory cacheDatabase replication
    Contributing factor
    Single point of failureRetry stormSignal buried in noise
    Detection

    A flood of alerts from PagerDuty, Nagios and Prometheus, with monitoring showing the lost metro link. A lingering alert for one Redis DB instance later pointed to the replication problem.

    Remediation
    • Turned chat off at the load balancer layer so the rest of BC3 could stabilise
    • Restarted Redis on the replica that was stuck in the resynchronization loop
    Action items
    • Make the documented Redis configuration changes for the known replication issue
    • Make concrete plans to remove the shared physical path under the two metro links
    • Prove the failover systems in production at peak load
    • Make explicit room for post-incident follow-up work and push other work out
    Lessons

    Planned redundancy and live testing had given Basecamp a false sense of security, especially at peak load: moving between redundant network links or shifting load across data centers should work without issue, and Basecamp would rather find out in a controlled test with the full team ready. The follow-up from an outage needs urgency and explicitly cleared time, not just a place on a list.

    View case
Check your cluster for this

These 2 cases involve API gateway and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure2 cases

Component
API gateway1Backbone network (WAN)1BGP routing1In-memory cache1CDN and edge network1Database replication1Amazon ECS1
Contributing factor
Retry storm2Signal buried in noise1Automation amplification1Unpatched known bug1Monitoring blind spot1Overlapping failures1Single point of failure1
Organisation
37signals1Canva1
  1. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task
    Canva2024
  2. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020