Cases2 cases

1 to 2 of 2

1. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task

   Canva2024

   Symptoms

   On 12 November 2024, after a new version of Canva's editor was deployed at 8:47 AM UTC, the editor's object panel stayed in a perpetual loading state for many users, particularly in Asia. Then from 9:08 AM UTC to about 10:00 AM UTC canva.com was unavailable and all requests to it failed.

   Service outageOOMKilledHigh latencyPacket loss

   Trigger

   At 8:47 AM UTC a new version of the Canva editor page was deployed, and client devices began fetching its new static assets through Cloudflare while the network path between Cloudflare's Singapore and Ashburn locations was losing packets.

   Root cause

   Dependency failure

   Component

   CDN and edge networkAPI gatewayAmazon ECS

   Contributing factor

   Overlapping failuresMonitoring blind spotRetry stormAutomation amplificationUnpatched known bug

   Remediation

   • Added a temporary Cloudflare firewall rule at 9:29 AM UTC that blocked all traffic at the CDN so new API Gateway tasks could start
   • Redirected canva.com to the status page
   • Restored traffic gradually, starting with Australian users under strict rate limits

   Action items

   • Write a runbook for rerouting, blocking and progressively restoring traffic
   • Show users a more informative error page when canva.com is unavailable
   • Raise the baseline number of API Gateway tasks and the memory of each task
   • Add load shedding rules to the API Gateway for similar traffic patterns
   • Load test the API Gateway regularly
   • Deploy the patch for the thread-locking bug in the telemetry library
   • Add multithreaded tests for the contended code path to the telemetry library microbenchmark suite
   • Use page load completion events as a second canary indicator for page asset releases
   • Experiment with adding another rollout stage to the canary to give more time to detect issues
   • Add timeouts so user requests do not wait excessively long for assets
   • Work with Cloudflare to understand the system interactions behind the incident

   View case
2. IL-0273 A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hours

   GitHub2016

   Symptoms

   At 00:23 UTC on 28 January 2016 GitHub became unavailable and returned HTTP 503 responses with the unicorn error page. On-call engineers received alerts for unreachable servers and a spike in exceptions related to Redis connectivity, after slightly over 25% of the servers and several network devices in the primary data center rebooted.

   Service outage

   Trigger

   At 00:23 UTC a brief disruption in the systems that supply power to the servers and equipment of GitHub's primary data center rebooted slightly over 25% of the servers and several network devices.

   Root cause

   Dependency failure

   Component

   In-memory cacheData center power and cooling

   Contributing factor

   Unpatched known bugSingle point of failureGap in automated testsCircular dependency

   Detection

   The reboots generated alerts to several on-call engineers, starting with notifications for unreachable servers and a spike in exceptions related to Redis connectivity.

   Remediation

   • Worked with on-site facilities technicians to drain flea power from the failed servers and start them cold so their disks were visible
   • Rebuilt the affected Redis clusters on alternate hardware
   • Restored the Redis cluster data onto standby equipment and brought the Redis-server processes back online
   • Removed the maintenance page after the HAProxy load balancers returned the recovered application servers to the pool

   Action items

   • Update the server firmware with the known issue across the fleet
   • Make tooling open issues automatically when new firmware updates are available, so their changelogs are reviewed
   • Add tests that make sure application processes start even when certain external systems are unavailable
   • Improve circuit breakers so functionality degrades gracefully when backend services are down
   • Review the availability requirements of internal systems such as server provisioning so they are on par with user-facing systems
   • Strengthen communication between teams during incidents
   • Predefine escalation strategies for all-hands incidents
   • Improve status messaging to users during incidents

   Lessons

   Dependencies between components fail in new ways unless they are tested, and a power loss cannot be fully prevented, so the goal is fast and reliable recovery. The application should start without non-essential backends, and internal systems needed to recover from an outage, such as server provisioning, must be as reliable as the system being recovered.

   View case

By root cause1