IL-0410 AssemblyAI · 2026-09-16awsai-infra

SQS Fair Queues rejected valid messages with InvalidParameterValue, failing about half of AssemblyAI's US async requests

AssemblyAI's transcription pipeline distributes work through AWS SQS queues. On 16 September 2026 the SQS Fair Queues feature started rejecting valid messages with InvalidParameterValue although nothing had changed on AssemblyAI's side, and requests that could not be queued failed. AssemblyAI posted the issue as identified at 20:24 UTC, affecting about 50% of async transcriptions on the US endpoint, and mitigated by disabling Fair Queues across its services and going back to standard SQS queues. Errors began to fall at 21:10 and the Async API was recovering at 21:12; the elevated errors lasted from 20:04 to 21:08 UTC, roughly an hour. AWS confirmed a service-side issue and rolled back its change later that evening. Much of AssemblyAI's service configuration lives in environment variables that only take effect after task restarts, and AssemblyAI is moving it to a global feature flag system so that mitigations like this one take seconds rather than minutes. No customer audio or transcript data was lost, and failed requests can be resubmitted.

Severity: –
Scope: US Async API: at peak roughly 50% of async transcription requests failed with server errors, and some successful requests were slower
Users affected: Customers using the Async API on the US endpoint
Region: US endpoint; EU endpoint and real-time/streaming services not affected
Time to detect: –
Time to resolve: 1h 4m

Symptoms

From 20:04 UTC on 16 September 2026 AssemblyAI's US Async API returned elevated server errors. At the peak roughly 50% of async transcription requests on the US endpoint failed, and part of the requests that succeeded took longer than usual to complete.

The errors came from AWS SQS, the queue service the transcription pipeline uses to hand out work: the Fair Queues feature began rejecting valid messages with an InvalidParameterValue error on a parameter AssemblyAI's services had sent successfully for months. The queues sit between the API and the transcription workers, so requests that could not be queued failed outright, and part of the work that got through was delayed behind reduced throughput. No deploy or infrastructure change on AssemblyAI's side came before it. The EU endpoint and the real-time and streaming services were not affected.

Symptom High error rate High latency

Root cause

Root cause Cloud provider outage

Component Message broker

Contributing factor –

Trigger: The AWS SQS Fair Queues feature began rejecting valid messages with InvalidParameterValue because of a change on the AWS side; AWS confirmed a service-side issue.

Detection

–

Remediation

Disabled Fair Queues across all services and reverted to standard SQS queues

Action items

Move service configuration from environment variables to a global feature flag system so mitigations apply in seconds
Add a reviewed break-glass path that lets urgent fixes skip non-essential CI steps
Fail the US pipeline over to a second region when a regional provider dependency degrades

Lessons

–

Sources

https://status.assemblyai.com/incidents/jfgvq10x906m · official-status

Related cases

IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeperAppOptics · 2018
same root cause · shared Message broker

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Message broker among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0410
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0410/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.