Cases4 cases

1 to 4 of 4

1. IL-0410 SQS Fair Queues rejected valid messages with InvalidParameterValue, failing about half of AssemblyAI's US async requests

   AssemblyAI2026

   Symptoms

   From 20:04 UTC on 16 September 2026 AssemblyAI's US Async API returned elevated server errors. At the peak roughly 50% of async transcription requests on the US endpoint failed, and part of the requests that succeeded took longer than usual to complete.

   High error rateHigh latency

   Trigger

   The AWS SQS Fair Queues feature began rejecting valid messages with InvalidParameterValue because of a change on the AWS side; AWS confirmed a service-side issue.

   Root cause

   Cloud provider outage

   Component

   Message broker

   Contributing factor

   –

   Remediation

   • Disabled Fair Queues across all services and reverted to standard SQS queues

   Action items

   • Move service configuration from environment variables to a global feature flag system so mitigations apply in seconds
   • Add a reviewed break-glass path that lets urgent fixes skip non-essential CI steps
   • Fail the US pipeline over to a second region when a regional provider dependency degrades

   View case
2. IL-0390 PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalled

   PostHog2025

   Symptoms

   From 11 November 2025 at 16:02 UTC, PostHog's event ingestion pipeline for US Cloud became severely degraded, and events for all customers in the US region were processed with delays from 10 minutes up to 2 days. Writes to the Postgres database that stores Person information could take multiple seconds per update, and the pipeline slowed until no standard scaling option helped.

   Throughput degradationHigh latency

   Trigger

   The 32-bit OID space used to track TOASTed values of the Persons table's properties column, a JSONb field that often exceeds the 2KB threshold for out-of-line storage, was nearly exhausted.

   Root cause

   Capacity exhaustion

   Component

   DatabaseMessage broker

   Contributing factor

   Unknown limitMonitoring blind spotNo runbook

   Remediation

   • Created a new partitioned Persons table with a fresh TOAST table and OID space
   • Added database triggers to keep the old and new Persons tables in sync for live writes
   • Copied the existing Person data from the old table to the new one with a backfill script
   • Changed ingestion and application code to write to the new table and read from it first, with fallback to the old table
   • Deployed the web app changes to production under manual control rather than a fully automated rollout
   • Scaled ingestion workers to process the accumulated backlog
   • Paused ingestion when the MSK local disk reached about 85%
   • Reduced the local retention of the affected Kafka topic
   • Resumed ingestion after confirming there was enough disk headroom
   • Moved the backfill into Dagster to finish it over the weekend

   Action items

   • Add metrics and alerts for heavy disk read amplification per write
   • Add metrics and alerts for TOAST table statistics
   • Add metrics and alerts for other Postgres engine limits that matter at large scale
   • Write runbooks for engine-level Postgres limits, with decision trees for migrating versus repairing in place
   • Create customer communication runbooks with a clear escalation path and redundancies
   • Evaluate other data stores for the persons database
   • Complete the reconsolidation on the new Persons tables
   • Improve the backfill tooling and the safeguards for Kafka tiered storage
   • Design and operate Person-like tables so they avoid similar limits

   Lessons

   TOAST OID exhaustion degrades writes long before the hard limit of about 4 billion OIDs, it is rare even at large scale, and standard dashboards for CPU, memory, IOPS and lock contention did not immediately point at it, so diagnosis took a day and a half without monitoring of TOAST growth, OID usage or disk reads per write. Many core features depended on timely updates to the persons table, so its slowdown reached a wide part of the product. The first backfill approach lacked visibility for a long migration, the Kafka tiered storage settings and alerts were not tuned for a large backlog, and the people who normally handle customer communications were all unavailable.

   View case
3. IL-0535 An internal Kinesis Data Streams cell in US-EAST-1 degraded, raising errors in CloudWatch Logs, S3 events and Lambda

   Amazon Web Services2024

   Symptoms

   From 2:45 PM PDT on July 30th, 2024, several AWS services in the Northern Virginia (US-EAST-1) Region had higher latency and error rates: CloudWatch Logs, Amazon Data Firehose, the event framework that publishes Amazon S3 events, Amazon ECS, AWS Lambda, Amazon Redshift and AWS Glue. All of them used one internal cell of Amazon Kinesis Data Streams, whose requests had started to see elevated latency and errors.

   High latencyHigh error rateMonitoring outageProbe failures

   Trigger

   At 9:09 AM PDT on July 30th a routine deployment started in one Availability Zone of the Kinesis cell, and taking hosts in and out of service during the deployment set off the failure.

   Root cause

   Software defect

   Component

   Message brokerServerless function platformObject storage

   Contributing factor

   Unrepresentative stagingAutomation amplification

   Remediation

   • Deployed a change that shed load by reducing incoming stream volume from less time-sensitive internal workloads
   • Added capacity to the subsystem that provisions secure connections for Kinesis requests
   • Added more capacity at 6:32 PM PDT to further improve connection provisioning

   Action items

   • Keep the added connection provisioning capacity, load-shedding tooling and changed connection limits in place in US-EAST-1
   • Apply the same changes in every other region that has cells with the new Kinesis Data Streams architecture
   • Update the cell management system to better manage the workload that triggered the issue

   View case
4. IL-0384 Amazon Kinesis front-end servers in US-EAST-1 could not route requests, so Kinesis and services built on it failed

   Amazon Web Services2020

   Symptoms

   At 5:15 AM PST on 25 November 2020, alarms fired for errors on putting and getting Amazon Kinesis records in the Northern Virginia (US-EAST-1) Region. From the same time CloudWatch, which processes metric and log data through Kinesis Data Streams, and CloudWatch Events and EventBridge returned more errors.

   High error rateHigh latencyMonitoring outage

   Trigger

   A relatively small addition of capacity to the Kinesis front-end fleet, made between 2:44 AM and 3:47 AM PST.

   Root cause

   Capacity exhaustion

   Component

   Message brokerServerless function platformTelemetry agent

   Contributing factor

   Unknown limitExcessive blast radiusCircular dependency

   Detection

   Alarms for errors on putting and getting Kinesis records fired at 5:15 AM PST, and the teams began reviewing logs.

   Remediation

   • Removed the newly added front-end capacity
   • Added a configuration so front-end servers load bootstrap data from the authoritative metadata store instead of neighbouring servers
   • Restarted the front-end fleet gradually, a few hundred servers per hour from 10:07 AM PST
   • Added Cognito capacity to buffer more calls to Kinesis
   • Deployed a change that reduced Cognito's dependency on Kinesis
   • Mitigated the memory contention on the Lambda invocation hosts
   • Informed customers through the backup Service Health Dashboard tool, the Personal Health Dashboard and a global banner

   Action items

   • Move the Kinesis front-end to larger CPU and memory servers so fewer servers, and fewer threads per server, are needed
   • Add fine-grained alarming for thread consumption
   • Finish testing a higher operating system thread count limit
   • Greatly improve the cold-start time of the front-end fleet
   • Move the front-end server cache to a dedicated fleet
   • Move a few large AWS services such as CloudWatch to a separate, partitioned front-end fleet
   • Accelerate the cellularization of the front-end fleet to match the back-end
   • Change the Cognito webservers so they sustain Kinesis API errors without exhausting their buffers
   • Keep 3 hours of metric data in the CloudWatch local metrics data store, first in US-EAST-1 and then globally
   • Train support engineers regularly on the backup tool for posting to the Service Health Dashboard

   Lessons

   In the Kinesis front-end, the number of threads each server keeps grows directly with the number of servers in the fleet, so adding servers can push every server past an operating system limit at the same time. AWS sees cellularization of the front-end, already done for the back-end, as the way to keep the shard-map cache within a tested and operated range and to guard against unknown scaling limits.

   View case

By root cause3