Incident Library
Applied filters Root causeCloud provider outageComponentMessage brokerClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0410 SQS Fair Queues rejected valid messages with InvalidParameterValue, failing about half of AssemblyAI's US async requests
    AssemblyAI2026
    Symptoms

    From 20:04 UTC on 16 September 2026 AssemblyAI's US Async API returned elevated server errors. At the peak roughly 50% of async transcription requests on the US endpoint failed, and part of the requests that succeeded took longer than usual to complete.

    High error rateHigh latency
    Trigger

    The AWS SQS Fair Queues feature began rejecting valid messages with InvalidParameterValue because of a change on the AWS side; AWS confirmed a service-side issue.

    Root cause
    Cloud provider outage
    Component
    Message broker
    Contributing factor
    –
    Remediation
    • Disabled Fair Queues across all services and reverted to standard SQS queues
    Action items
    • Move service configuration from environment variables to a global feature flag system so mitigations apply in seconds
    • Add a reviewed break-glass path that lets urgent fixes skip non-essential CI steps
    • Fail the US pipeline over to a second region when a regional provider dependency degrades
    View case
  2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018
    Symptoms

    On May 31, 2018 from 21:53 UTC the metrics APIs of AppOptics and Librato returned a 5xx for a majority of incoming metric data. APM metrics and traces sent from AppOptics APM were affected, and many alert notifications were delayed.

    High error rateFailover failureMonitoring outage
    Trigger

    A power event in one of AWS's availability zones in us-east-1 cut off nodes in several critical parts of the AppOptics and Librato infrastructure, including the ZooKeeper node that was then the leader of the ensemble coordinating the Kafka brokers.

    Root cause
    Cloud provider outage
    Component
    Message brokerCoordination and key-value store
    Contributing factor
    Single point of failureExcessive blast radiusMonitoring blind spot
    Remediation
    • Restarted one ZooKeeper node by hand to force a leader election, after which the ensemble regained quorum
    • Restarted every Kafka broker in a rolling restart until all partitions were back online
    • Worked through the backlog, including the backlog of alerts, and resumed normal operations
    Action items
    • Isolate the Kafka clusters better across availability zones and regions
    • Stop sharing one ZooKeeper ensemble across Kafka clusters
    • Run game days on this failure and check whether newer Kafka versions fix it
    • Put in place more checks that notify the team earlier about subsystems that need extra help to recover
    Lessons

    Running services across several availability zones and more than one Kafka cluster did not isolate the pipeline from the failure of one zone, because partitions of different clusters sat in the same zones and the clusters shared one ZooKeeper ensemble. More operational monitoring of the subsystems that needed help to recover would have led to a faster recovery.

    View case
Check your cluster for this

These 2 cases involve Message broker and Coordination and key-value store. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage2 cases

Component
Message broker2Coordination and key-value store1
Contributing factor
Excessive blast radius1Monitoring blind spot1Single point of failure1
Organisation
AppOptics1AssemblyAI1
  1. IL-0410 SQS Fair Queues rejected valid messages with InvalidParameterValue, failing about half of AssemblyAI's US async requests
    AssemblyAI2026
  2. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeper
    AppOptics2018