Incident Library

IL-0409 AssemblyAI · 2026-01-29awsai-infra

AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero

AssemblyAI's traffic had been growing steadily when an influx of batch transcription jobs, larger than usual, arrived. Autoscaling added workers as intended: it ran out of spot instances, fell back to on-demand instances and then hit the AWS Elastic Network Interface (ENI) limit. Every ECS task needs an ENI. With no ENIs left, the authentication service, which shared resources with the batch workers, could not rotate its tasks and scaled to zero, and the Async Transcription API and LLM Gateway stopped. While the team tried to recover, a Lambda function built to help with failures flooded the AWS API with describe calls and got the operations needed for recovery rate-limited. After 25 minutes of outage and two hours of degraded performance, the backlog was cleared and turnaround times returned to normal. AssemblyAI says it did not monitor ENI utilisation, that the traffic spike on the batch side starved the authentication service through the shared resources, and that it had warning signs it did not connect.

Severity
–
Scope
Async Transcription API and LLM Gateway down for 25 minutes, then two hours of degraded performance with elevated turnaround times; website down and logins affected during recovery
Users affected
–
Region
NA API endpoints; streaming API and EU endpoint not affected
Time to detect
–
Time to resolve
2h 25m

Symptoms

On 29 January 2026 AssemblyAI's Async Transcription API and LLM Gateway on the NA API endpoints went down for 25 minutes, returning 500 errors, and then ran degraded for two hours. The status page first reported the investigation at 17:54 UTC.

As requests began to complete again, some still returned 500 errors, the website was down and logins could be affected. Completed requests had elevated turnaround times while a backlog of requests was worked off, and turnaround times were back to normal by 19:32 UTC. An update at 18:09 UTC said the streaming API and the EU endpoint were running normally.

Recovery was held back by a Lambda function meant to help with failures: it sent so many describe calls to the AWS API that the API rate-limited the very operations AssemblyAI needed to bring the services back.

Symptom Service outageHigh error rateHigh latencyCloud API rate limit exhausted

Root cause

Root cause Capacity exhaustion

Component Amazon ECS
Contributing factor Excessive blast radiusMonitoring blind spotAutomation amplificationIgnored early signal
Trigger
An influx of batch transcription jobs higher than usual made autoscaling add workers until spot instances ran out, on-demand instances were used, and the AWS Elastic Network Interface limit was reached.

Detection

–

Remediation

Action items

Lessons

AssemblyAI calls the outage preventable: ENI utilisation was not monitored, and because the authentication service shared resources with batch workers, a traffic spike in one starved the other. The warning signs were there but were not connected, and a tool built for recovery ended up blocking recovery.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Amazon ECS among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.