Incident Library

Applied filters Root causeCapacity exhaustionSymptomHigh latencyFactorIgnored early signalClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases4 cases

1 to 4 of 4

IL-0553 CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes
CircleCI2026
Symptoms
On 6 October 2026 CircleCI pipelines were slow to be created and workflows and jobs were slow to start, in short bursts from 13:30 UTC and continuously from 15:30 UTC. From 15:48 to 16:38 UTC no workflows or jobs started at all, and pipelines triggered from 16:02 UTC failed.
High latency Service outage Throughput degradation Node disk full
Trigger
Around 12:40 UTC, on top of the regular workload, anti-wraparound autovacuum runs on several of the largest tables of the orchestration service's database began writing transaction logs faster than the database could archive them.
Root cause
Capacity exhaustion
Component
Database PostgreSQL vacuum CI system
Contributing factor
Ignored early signal No alert on limit Single point of failure
Detection
Monitoring raised short alerts about errors in the orchestration service from 10:02 UTC, each clearing on its own. At 13:35 the alerts came back together with related alerts from several other services, and after they returned around 14:30 and engineers investigated further, an incident was declared at 14:54 UTC.
Remediation
Changed the anti-wraparound autovacuum settings to help it finish faster
Moved the transaction logs onto the database's main storage volume, which took the database offline from 15:47 to 16:38 UTC
Changed database parameters to stop further maintenance runs from starting
Changed a database setting at 18:19 UTC to cut the time each write spent waiting on storage
Added database and server capacity to the systems that hand jobs to the execution fleet to clear the backlog faster
Action items
Alert on how full the transaction log volume is and on how far archiving has fallen behind
Alert on the database maintenance that drives transaction log growth
Review all alerts so that noisy or sensitive alerts do not hide real issues
Tune how often and how aggressively autovacuum runs on the database
Track the size of the largest tables over time so they stay within safe limits
Remove the single transaction log bottleneck by splitting the service across several database instances or moving it to a distributed database
Make CircleCI's systems wait for the orchestration service to recover instead of failing jobs after retrying
Include specific impact and timing sooner in status page updates during long incidents
Lessons
CircleCI says the combination of alerts should have been recognised sooner: brief error alerts that cleared on their own were treated as routine, and nothing alerted on the transaction log volume or on archiving lag. Because every write of the orchestration service went through one database and one transaction log, a backlog in that log slowed every pipeline, workflow and job.
View case
IL-0409 AssemblyAI's async and LLM Gateway APIs returned 5xx after its auth service could not get ENIs and scaled to zero
AssemblyAI2026
Symptoms
On 29 January 2026 AssemblyAI's Async Transcription API and LLM Gateway on the NA API endpoints went down for 25 minutes, returning 500 errors, and then ran degraded for two hours. The status page first reported the investigation at 17:54 UTC.
Service outage High error rate High latency Cloud API rate limit exhausted
Trigger
An influx of batch transcription jobs higher than usual made autoscaling add workers until spot instances ran out, on-demand instances were used, and the AWS Elastic Network Interface limit was reached.
Root cause
Capacity exhaustion
Component
Amazon ECS
Contributing factor
Excessive blast radius Monitoring blind spot Automation amplification Ignored early signal
Remediation
Worked through the backlog of requests until turnaround times were back to normal
Action items
Add alerts on ENI utilisation
Run the authentication service on dedicated capacity, separate from batch workers
Fix the Lambda function that rate-limited the AWS API during recovery
Run everything in multiple regions
Lessons
AssemblyAI calls the outage preventable: ENI utilisation was not monitored, and because the authentication service shared resources with batch workers, a traffic spike in one starved the other. The warning signs were there but were not connected, and a tool built for recovery ended up blocking recovery.
View case
IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes
Discord2023
Symptoms
On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.
Service outage Database overload High latency
Trigger
During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.
Root cause
Capacity exhaustion
Component
Database
Contributing factor
No headroom Overlapping failures Dependence on a warm cache Unverified assumption Ignored early signal Retry storm
Detection
An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.
Remediation
Turned off sending typing events and marking messages as read to shed load from the authentication cluster
Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
Restarted the two zone [b] nodes one at a time
Action items
Double the size of the scylla-authentication cluster so each node takes half as many requests
Rebuild scylla-authentication-b-2 with new, non-degraded disks
Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region
Lessons
Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.
View case
IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
PagerDuty2014sev1
Symptoms
On 3 June 2014, PagerDuty's notification pipeline performed poorly and some notifications were delayed after the Cassandra cluster behind it became heavily degraded. About 3% of events sent to the integration API could not be received.
Database overload High error rate High latency
Trigger
On 3 June 2014 the Cassandra background repair process started its normal run on one node of the cluster while other high load was on the cluster; on 4 June the team restarted the postponed repair by hand.
Root cause
Capacity exhaustion
Component
Database
Contributing factor
No headroom Excessive blast radius Harmful manual action Ignored early signal
Remediation
Reduced the load on the Cassandra cluster
Stopped the Cassandra repair process
Cut communication between some Cassandra nodes to stabilise the cluster
On 4 June, deleted all in-flight data in the notification pipeline in a "factory reset"
Brought downstream systems back into a consistent state with the emptied pipeline by hand
Action items
Scale the existing Cassandra nodes vertically to bigger, faster servers
Set up multiple Cassandra clusters and spread the workloads across them
Set system load thresholds at which the Cassandra clusters are scaled out horizontally ahead of need
Upgrade the current and new clusters to a more recent Cassandra version
Add more load shedding techniques to control Cassandra under high load
Bring more Cassandra expertise in-house
Lessons
PagerDuty concluded that its Cassandra cluster was underscaled and shared by too many services with different load and usage patterns, and that the scaling work should have come before the planned Cassandra version upgrade.
View case

Check your cluster for this

These 4 cases involve Database and PostgreSQL vacuum. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion4 cases

Component: Database3 PostgreSQL vacuum1 CI system1 Amazon ECS1
Contributing factor: Ignored early signal4 Excessive blast radius2 No headroom2 Unverified assumption1 Automation amplification1 Harmful manual action1 Monitoring blind spot1 No alert on limit1 Overlapping failures1 Retry storm1 Single point of failure1 Dependence on a warm cache1
Organisation: AssemblyAI1 CircleCI1 Discord1 PagerDuty1