Incident Library

IL-0257 Canva · 2024-11-12networkingaws

canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task

Canva's editor is a single-page application deployed several times a day; each deployment publishes over 100 new static assets, including JavaScript files, to an AWS S3 bucket. Clients fetch them through Cloudflare's tiered cache, where local data centers ask regional or upper-tier data centers and those ask the S3 origin. The editor's JavaScript then calls the API Gateway, the entry point for all of Canva's API requests, which handles authentication, authorization and rate limiting, is built on Netty and runs as an autoscaled group of tasks on Amazon ECS.

According to Cloudflare, a stale rule in its traffic management system was sending user IPv6 traffic over public transit between Ashburn and Singapore instead of over its private backbone, and the rule kept Cloudflare's automation from routing around the packet loss. Cloudflare's cache stream feature (Concurrent Streaming Acceleration) consolidates user requests for the same asset into a single request, so more than 270,000 requests waited on one slow fetch and were released together at 9:07 AM UTC. The resulting 1.5 million requests per second met an API Gateway weakened by a performance regression in Canva's telemetry library: certain metrics were re-registered each time a value was recorded, under a lock in a third-party library, and the lock contention on the event loop threads sharply cut how much one task could handle. A fix for that bug had entered the release process the same day, but its impact had been underestimated and it was not expedited. Off-heap memory growth led the OOM killer to terminate all API Gateway containers, faster than autoscaling could replace them.

At 9:29 AM UTC Canva added a temporary Cloudflare firewall rule that blocked all traffic at the CDN, so new API Gateway tasks could start without being flooded, and later redirected canva.com to its status page. Once the number of healthy tasks was stable, Canva restored traffic step by step, starting with Australian users under strict rate limits. Cloudflare removed the traffic management rule that had sent requests over the lossy path.

Severity
–
Scope
canva.com was unavailable from 9:08 AM UTC to about 10:00 AM UTC; before that the editor's object panel did not load for many users
Users affected
–
Region
The object panel failure hit many users in Asia, and requests backed up from users in Southeast Asia
Time to detect
–
Time to resolve
52m

Symptoms

On 12 November 2024, after a new version of Canva's editor was deployed at 8:47 AM UTC, the editor's object panel stayed in a perpetual loading state for many users, particularly in Asia. Then from 9:08 AM UTC to about 10:00 AM UTC canva.com was unavailable and all requests to it failed.

The object panel was waiting for one JavaScript chunk delivered through Cloudflare. The network path between Cloudflare's Singapore (SIN) and Ashburn (IAD) locations had latency problems: the 90th percentile time-for-first-byte rose by over 1700%, packet loss on the path reached about 66% at peak, and the fetch of that one asset took 20 minutes. Canva's canary system, which normally aborts a deployment when errors rise, recorded no errors because the requests never completed, while more than 270,000 user requests for the file waited on the same Cloudflare cache stream.

At 9:07 AM UTC the fetch completed and all waiting requests finished at once. The object panel loaded on every waiting device together, sending 1.5 million requests per second to the API Gateway, three times the typical peak. Requests backed up, the load balancers opened new connections to the already overloaded tasks, off-heap memory grew, and the Linux Out Of Memory Killer terminated all running API Gateway containers within the first 2 minutes.

Autoscaling started replacement tasks, but each new task was overwhelmed by the traffic as soon as it was marked healthy and was terminated. Raising the desired task count by hand did not stop the new tasks from being terminated.

Symptom Service outageOOMKilledHigh latencyPacket loss

Root cause

Root cause Dependency failure

Component CDN and edge networkAPI gatewayAmazon ECS
Contributing factor Overlapping failuresMonitoring blind spotRetry stormAutomation amplificationUnpatched known bug
Trigger
At 8:47 AM UTC a new version of the Canva editor page was deployed, and client devices began fetching its new static assets through Cloudflare while the network path between Cloudflare's Singapore and Ashburn locations was losing packets.

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CDN and edge network, API gateway, Amazon ECS among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.