Cases3 cases

1 to 3 of 3

1. IL-0409 AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero

   AssemblyAI2026

   Symptoms

   On 29 January 2026 AssemblyAI's Async Transcription API and LLM Gateway on the NA API endpoints went down for 25 minutes, returning 500 errors, and then ran degraded for two hours. The status page first reported the investigation at 17:54 UTC.

   Service outageHigh error rateHigh latencyCloud API rate limit exhausted

   Trigger

   An influx of batch transcription jobs higher than usual made autoscaling add workers until spot instances ran out, on-demand instances were used, and the AWS Elastic Network Interface limit was reached.

   Root cause

   Capacity exhaustion

   Component

   Amazon ECS

   Contributing factor

   Excessive blast radiusMonitoring blind spotAutomation amplificationIgnored early signal

   Remediation

   • Worked through the backlog of requests until turnaround times were back to normal

   Action items

   • Add alerts on ENI utilisation
   • Run the authentication service on dedicated capacity, separate from batch workers
   • Fix the Lambda function that rate-limited the AWS API during recovery
   • Run everything in multiple regions

   Lessons

   AssemblyAI calls the outage preventable: ENI utilisation was not monitored, and because the authentication service shared resources with batch workers, a traffic spike in one starved the other. The warning signs were there but were not connected, and a tool built for recovery ended up blocking recovery.

   View case
2. IL-0257 canva.com went down for about 50 minutes when 1.5 million requests per second OOM-killed every API Gateway task

   Canva2024

   Symptoms

   On 12 November 2024, after a new version of Canva's editor was deployed at 8:47 AM UTC, the editor's object panel stayed in a perpetual loading state for many users, particularly in Asia. Then from 9:08 AM UTC to about 10:00 AM UTC canva.com was unavailable and all requests to it failed.

   Service outageOOMKilledHigh latencyPacket loss

   Trigger

   At 8:47 AM UTC a new version of the Canva editor page was deployed, and client devices began fetching its new static assets through Cloudflare while the network path between Cloudflare's Singapore and Ashburn locations was losing packets.

   Root cause

   Dependency failure

   Component

   CDN and edge networkAPI gatewayAmazon ECS

   Contributing factor

   Overlapping failuresMonitoring blind spotRetry stormAutomation amplificationUnpatched known bug

   Remediation

   • Added a temporary Cloudflare firewall rule at 9:29 AM UTC that blocked all traffic at the CDN so new API Gateway tasks could start
   • Redirected canva.com to the status page
   • Restored traffic gradually, starting with Australian users under strict rate limits

   Action items

   • Write a runbook for rerouting, blocking and progressively restoring traffic
   • Show users a more informative error page when canva.com is unavailable
   • Raise the baseline number of API Gateway tasks and the memory of each task
   • Add load shedding rules to the API Gateway for similar traffic patterns
   • Load test the API Gateway regularly
   • Deploy the patch for the thread-locking bug in the telemetry library
   • Add multithreaded tests for the contended code path to the telemetry library microbenchmark suite
   • Use page load completion events as a second canary indicator for page asset releases
   • Experiment with adding another rollout stage to the canary to give more time to detect issues
   • Add timeouts so user requests do not wait excessively long for assets
   • Work with Cloudflare to understand the system interactions behind the incident

   View case
3. IL-0446 About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deploy

   ITANDI2020

   Symptoms

   Right after a deploy to the production ECS (Fargate) service, HTTPCode_ELB_5XX_Count on the service's ALB jumped. The service receives about 10,000 requests a minute, and roughly 20% of them got a 5XX.

   High error rate

   Trigger

   A deploy recreated the ECS tasks, and at that moment communication between the ALB and its backends became unstable.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerAmazon ECS

   Contributing factor

   –

   Detection

   The team saw HTTPCode_ELB_5XX_Count jump on the ALB right after the deploy, together with target connection errors.

   Remediation

   • Restarted the ECS tasks, which did not help
   • Moved the ALB's ap-northeast-1d subnet to another subnet in the same zone, which did not help
   • Moved the six ECS tasks running in 1d to 1a and 1c, which did not help
   • Removed the 1d subnet from the ALB, which did not help
   • Changed the ALB idle timeout, which did not help
   • Rolled the ECS tasks back to the version before the deploy, which did not help
   • Created a new ALB with the same VPC, subnets, backends and security groups
   • Attached both ALBs to the ECS service
   • Sent one tenth of the traffic to the new ALB with a weighted DNS record
   • Moved all traffic to the new ALB after it returned no 5XX

   Lessons

   When a graph shows something that should be impossible on the network, suspect the AWS side as well.

   View case

By root cause3