Incident Library
Applied filters Root causeCapacity exhaustionSymptomHigh error rateFactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0409 AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero
    AssemblyAI2026
    Symptoms

    On 29 January 2026 AssemblyAI's Async Transcription API and LLM Gateway on the NA API endpoints went down for 25 minutes, returning 500 errors, and then ran degraded for two hours. The status page first reported the investigation at 17:54 UTC.

    Service outageHigh error rateHigh latencyCloud API rate limit exhausted
    Trigger

    An influx of batch transcription jobs higher than usual made autoscaling add workers until spot instances ran out, on-demand instances were used, and the AWS Elastic Network Interface limit was reached.

    Root cause
    Capacity exhaustion
    Component
    Amazon ECS
    Contributing factor
    Excessive blast radiusMonitoring blind spotAutomation amplificationIgnored early signal
    Remediation
    • Worked through the backlog of requests until turnaround times were back to normal
    Action items
    • Add alerts on ENI utilisation
    • Run the authentication service on dedicated capacity, separate from batch workers
    • Fix the Lambda function that rate-limited the AWS API during recovery
    • Run everything in multiple regions
    Lessons

    AssemblyAI calls the outage preventable: ENI utilisation was not monitored, and because the authentication service shared resources with batch workers, a traffic spike in one starved the other. The warning signs were there but were not connected, and a tool built for recovery ended up blocking recovery.

    View case
  2. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
    PagerDuty2014sev1
    Symptoms

    On 3 June 2014, PagerDuty's notification pipeline performed poorly and some notifications were delayed after the Cassandra cluster behind it became heavily degraded. About 3% of events sent to the integration API could not be received.

    Database overloadHigh error rateHigh latency
    Trigger

    On 3 June 2014 the Cassandra background repair process started its normal run on one node of the cluster while other high load was on the cluster; on 4 June the team restarted the postponed repair by hand.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No headroomExcessive blast radiusHarmful manual actionIgnored early signal
    Remediation
    • Reduced the load on the Cassandra cluster
    • Stopped the Cassandra repair process
    • Cut communication between some Cassandra nodes to stabilise the cluster
    • On 4 June, deleted all in-flight data in the notification pipeline in a "factory reset"
    • Brought downstream systems back into a consistent state with the emptied pipeline by hand
    Action items
    • Scale the existing Cassandra nodes vertically to bigger, faster servers
    • Set up multiple Cassandra clusters and spread the workloads across them
    • Set system load thresholds at which the Cassandra clusters are scaled out horizontally ahead of need
    • Upgrade the current and new clusters to a more recent Cassandra version
    • Add more load shedding techniques to control Cassandra under high load
    • Bring more Cassandra expertise in-house
    Lessons

    PagerDuty concluded that its Cassandra cluster was underscaled and shared by too many services with different load and usage patterns, and that the scaling work should have come before the planned Cassandra version upgrade.

    View case
Check your cluster for this

These 2 cases involve Database and Amazon ECS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Database1Amazon ECS1
Contributing factor
Excessive blast radius2Ignored early signal2Automation amplification1Harmful manual action1Monitoring blind spot1No headroom1
Organisation
AssemblyAI1PagerDuty1
  1. IL-0409 AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero
    AssemblyAI2026
  2. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
    PagerDuty2014