Cases2 cases

1 to 2 of 2

1. IL-0409 AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero

   AssemblyAI2026

   Symptoms

   On 29 January 2026 AssemblyAI's Async Transcription API and LLM Gateway on the NA API endpoints went down for 25 minutes, returning 500 errors, and then ran degraded for two hours. The status page first reported the investigation at 17:54 UTC.

   Service outageHigh error rateHigh latencyCloud API rate limit exhausted

   Trigger

   An influx of batch transcription jobs higher than usual made autoscaling add workers until spot instances ran out, on-demand instances were used, and the AWS Elastic Network Interface limit was reached.

   Root cause

   Capacity exhaustion

   Component

   Amazon ECS

   Contributing factor

   Excessive blast radiusMonitoring blind spotAutomation amplificationIgnored early signal

   Remediation

   • Worked through the backlog of requests until turnaround times were back to normal

   Action items

   • Add alerts on ENI utilisation
   • Run the authentication service on dedicated capacity, separate from batch workers
   • Fix the Lambda function that rate-limited the AWS API during recovery
   • Run everything in multiple regions

   Lessons

   AssemblyAI calls the outage preventable: ENI utilisation was not monitored, and because the authentication service shared resources with batch workers, a traffic spike in one starved the other. The warning signs were there but were not connected, and a tool built for recovery ended up blocking recovery.

   View case
2. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers

   CircleCI2021

   Symptoms

   From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

   Throughput degradationCloud API rate limit exhaustedDatabase overload

   Trigger

   The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

   Root cause

   Breaking change in an upgrade

   Component

   Message brokerDatabase

   Contributing factor

   Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures

   Detection

   Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

   Remediation

   • Blocked Linux machine jobs from entering the work queue
   • Had GCP raise the CPU quotas
   • Deleted old VMs by hand
   • Cancelled some workflows to free resources
   • Reset the VM services deployment so the VM destroy queues had consumers again
   • Scaled up the RDS database instance, at 18:20 and again at 19:15
   • Turned down VM provisioning and blocked remote-docker jobs to shed load
   • Scaled down all pods that created and destroyed VMs
   • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
   • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes

   Action items

   • Make VM provisioning more robust to changes in load so it can recover from thundering herds
   • Make VM provisioning independent so its failure does not affect Docker jobs
   • Review and improve observability of RabbitMQ and the production queues
   • Batch requests to avoid hitting GCP API rate limits
   • Improve incident response for long-running incidents

   View case

By root cause2