Cases2 cases

1 to 2 of 2

1. IL-0410 SQS Fair Queues rejected valid messages with InvalidParameterValue, failing about half of AssemblyAI's US async requests

   AssemblyAI2026

   Symptoms

   From 20:04 UTC on 16 September 2026 AssemblyAI's US Async API returned elevated server errors. At the peak roughly 50% of async transcription requests on the US endpoint failed, and part of the requests that succeeded took longer than usual to complete.

   High error rateHigh latency

   Trigger

   The AWS SQS Fair Queues feature began rejecting valid messages with InvalidParameterValue because of a change on the AWS side; AWS confirmed a service-side issue.

   Root cause

   Cloud provider outage

   Component

   Message broker

   Contributing factor

   –

   Remediation

   • Disabled Fair Queues across all services and reverted to standard SQS queues

   Action items

   • Move service configuration from environment variables to a global feature flag system so mitigations apply in seconds
   • Add a reviewed break-glass path that lets urgent fixes skip non-essential CI steps
   • Fail the US pipeline over to a second region when a regional provider dependency degrades

   View case
2. IL-0409 AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero

   AssemblyAI2026

   Symptoms

   On 29 January 2026 AssemblyAI's Async Transcription API and LLM Gateway on the NA API endpoints went down for 25 minutes, returning 500 errors, and then ran degraded for two hours. The status page first reported the investigation at 17:54 UTC.

   Service outageHigh error rateHigh latencyCloud API rate limit exhausted

   Trigger

   An influx of batch transcription jobs higher than usual made autoscaling add workers until spot instances ran out, on-demand instances were used, and the AWS Elastic Network Interface limit was reached.

   Root cause

   Capacity exhaustion

   Component

   Amazon ECS

   Contributing factor

   Excessive blast radiusMonitoring blind spotAutomation amplificationIgnored early signal

   Remediation

   • Worked through the backlog of requests until turnaround times were back to normal

   Action items

   • Add alerts on ENI utilisation
   • Run the authentication service on dedicated capacity, separate from batch workers
   • Fix the Lambda function that rate-limited the AWS API during recovery
   • Run everything in multiple regions

   Lessons

   AssemblyAI calls the outage preventable: ENI utilisation was not monitored, and because the authentication service shared resources with batch workers, a traffic spike in one starved the other. The warning signs were there but were not connected, and a tool built for recovery ended up blocking recovery.

   View case

By root cause2