Cases2 cases

1 to 2 of 2

1. IL-0409 AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero

   AssemblyAI2026

   Symptoms

   On 29 January 2026 AssemblyAI's Async Transcription API and LLM Gateway on the NA API endpoints went down for 25 minutes, returning 500 errors, and then ran degraded for two hours. The status page first reported the investigation at 17:54 UTC.

   Service outageHigh error rateHigh latencyCloud API rate limit exhausted

   Trigger

   An influx of batch transcription jobs higher than usual made autoscaling add workers until spot instances ran out, on-demand instances were used, and the AWS Elastic Network Interface limit was reached.

   Root cause

   Capacity exhaustion

   Component

   Amazon ECS

   Contributing factor

   Excessive blast radiusMonitoring blind spotAutomation amplificationIgnored early signal

   Remediation

   • Worked through the backlog of requests until turnaround times were back to normal

   Action items

   • Add alerts on ENI utilisation
   • Run the authentication service on dedicated capacity, separate from batch workers
   • Fix the Lambda function that rate-limited the AWS API during recovery
   • Run everything in multiple regions

   Lessons

   AssemblyAI calls the outage preventable: ENI utilisation was not monitored, and because the authentication service shared resources with batch workers, a traffic spike in one starved the other. The warning signs were there but were not connected, and a tool built for recovery ended up blocking recovery.

   View case
2. IL-0446 About 20% of requests to an ITANDI ECS service got ALB 5XX errors, apparently from one zone, right after a deploy

   ITANDI2020

   Symptoms

   Right after a deploy to the production ECS (Fargate) service, HTTPCode_ELB_5XX_Count on the service's ALB jumped. The service receives about 10,000 requests a minute, and roughly 20% of them got a 5XX.

   High error rate

   Trigger

   A deploy recreated the ECS tasks, and at that moment communication between the ALB and its backends became unstable.

   Root cause

   Cloud provider outage

   Component

   Cloud load balancerAmazon ECS

   Contributing factor

   –

   Detection

   The team saw HTTPCode_ELB_5XX_Count jump on the ALB right after the deploy, together with target connection errors.

   Remediation

   • Restarted the ECS tasks, which did not help
   • Moved the ALB's ap-northeast-1d subnet to another subnet in the same zone, which did not help
   • Moved the six ECS tasks running in 1d to 1a and 1c, which did not help
   • Removed the 1d subnet from the ALB, which did not help
   • Changed the ALB idle timeout, which did not help
   • Rolled the ECS tasks back to the version before the deploy, which did not help
   • Created a new ALB with the same VPC, subnets, backends and security groups
   • Attached both ALBs to the ECS service
   • Sent one tenth of the traffic to the new ALB with a weighted DNS record
   • Moved all traffic to the new ALB after it returned no 5XX

   Lessons

   When a graph shows something that should be impossible on the network, suspect the AWS side as well.

   View case

By root cause2