Cases5 cases

1 to 5 of 5

1. IL-0553 CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes

   CircleCI2026

   Symptoms

   On 6 October 2026 CircleCI pipelines were slow to be created and workflows and jobs were slow to start, in short bursts from 13:30 UTC and continuously from 15:30 UTC. From 15:48 to 16:38 UTC no workflows or jobs started at all, and pipelines triggered from 16:02 UTC failed.

   High latencyService outageThroughput degradationNode disk full

   Trigger

   Around 12:40 UTC, on top of the regular workload, anti-wraparound autovacuum runs on several of the largest tables of the orchestration service's database began writing transaction logs faster than the database could archive them.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuumCI system

   Contributing factor

   Ignored early signalNo alert on limitSingle point of failure

   Detection

   Monitoring raised short alerts about errors in the orchestration service from 10:02 UTC, each clearing on its own. At 13:35 the alerts came back together with related alerts from several other services, and after they returned around 14:30 and engineers investigated further, an incident was declared at 14:54 UTC.

   Remediation

   • Changed the anti-wraparound autovacuum settings to help it finish faster
   • Moved the transaction logs onto the database's main storage volume, which took the database offline from 15:47 to 16:38 UTC
   • Changed database parameters to stop further maintenance runs from starting
   • Changed a database setting at 18:19 UTC to cut the time each write spent waiting on storage
   • Added database and server capacity to the systems that hand jobs to the execution fleet to clear the backlog faster

   Action items

   • Alert on how full the transaction log volume is and on how far archiving has fallen behind
   • Alert on the database maintenance that drives transaction log growth
   • Review all alerts so that noisy or sensitive alerts do not hide real issues
   • Tune how often and how aggressively autovacuum runs on the database
   • Track the size of the largest tables over time so they stay within safe limits
   • Remove the single transaction log bottleneck by splitting the service across several database instances or moving it to a distributed database
   • Make CircleCI's systems wait for the orchestration service to recover instead of failing jobs after retrying
   • Include specific impact and timing sooner in status page updates during long incidents

   Lessons

   CircleCI says the combination of alerts should have been recognised sooner: brief error alerts that cleared on their own were treated as routine, and nothing alerted on the transaction log volume or on archiving lag. Because every write of the orchestration service went through one database and one transaction log, a backlog in that log slowed every pipeline, workflow and job.

   View case
2. IL-0288 ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipeline

   CDN772026

   Symptoms

   At 13:30 UTC on 9 March 2026, CDN77's data team was alerted to a sudden rise in the ZooKeeperWait metric on the ClickHouse cluster behind one of its large data pipelines. Alerting for this cluster size is set at 10 sec for the low watermark and 20 sec for the high one, and the metric reached thousands of seconds. The same rise appeared on three symmetrically redundant ClickHouse instances in different datacenters, each with its own ZooKeeper instance, and the pipeline stayed degraded for over an hour, with no observable impact on internal users, customers or partners.

   Replication lagThroughput degradation

   Trigger

   From 13:15 UTC two client machines that had been cut off from the network for several months came back and their log shippers sent months of queued, very small batches of health check data with old timestamps, spanning a very large time range.

   Root cause

   Capacity exhaustion

   Component

   DatabaseCoordination and key-value storeDatabase replicationTelemetry agent

   Contributing factor

   Monitoring blind spotUnreviewed defaultsIgnored early signal

   Detection

   An alert on the ZooKeeperWait metric fired at 13:30 UTC, fifteen minutes after the problematic inserts began at 13:15. The cluster's alerting uses a low watermark of 10 sec and a high watermark of 20 sec.

   Remediation

   • Dropped and recreated the one table partitioned by 5 minutes that had not cleared its backlog of parts
   • Broadened that table's partitioning from minutes to hours

   Action items

   • Reject out-of-range timestamps in the root MaterializedView tables
   • Record statistics on rejected rows in a second materialized view
   • Refine the deliberately wide rejection boundaries as data on rejections comes in
   • Keep a minimal test scenario with an insert query whose timestamps span days or months
   • Make a range of observability improvements

   Lessons

   Data structure matters even in a pipeline built around volume: incoherent timestamps can destabilise it as effectively as changes in cardinality. A metric that looks odd and cannot be explained at once should not be set aside under incident pressure; the falling mean insert size was what led to the cause. Means and medians each have their place, and here the mean, not the median, exposed the problem. Partitioning has to be chosen carefully: too few partitions give oversized parts, slow mutations and costly or ineffective TTL, while too many slow column scans, add final merging to aggregating queries and overload the Keeper.

   View case
3. IL-0270 Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutes

   Discord2023

   Symptoms

   On 6 November 2023 at 10:52 AM PST, when an upgrade script drained all connections from zone [c] of the scylla-authentication ScyllaDB cluster, Discord's availability, measured as the share of HTTP requests that return a 200-level response, fell at once to about 25%. Users could not connect, log in, read or send messages.

   Service outageDatabase overloadHigh latency

   Trigger

   During an OS upgrade that takes one zone of a ScyllaDB cluster offline at a time, the upgrade script drained all connections from zone [c] of scylla-authentication at 10:52 AM, while one node in the just-upgraded zone [b] had a degraded NVMe Local SSD.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomOverlapping failuresDependence on a warm cacheUnverified assumptionIgnored early signalRetry storm

   Detection

   An incident response call had already started when the first alerts fired at 10:56 AM, four minutes after the start. The time spent by downstream service graph showed the authentication service spending far more time on user requests than normal, and the authentication alerts fired moments later.

   Remediation

   • Turned off sending typing events and marking messages as read to shed load from the authentication cluster
   • Waited for the zone [c] RAID rebuild instead of restarting nodes, and the two zone [c] nodes came back online at 11:41 AM
   • Restarted the two zone [b] nodes one at a time

   Action items

   • Double the size of the scylla-authentication cluster so each node takes half as many requests
   • Rebuild scylla-authentication-b-2 with new, non-degraded disks
   • Build a way to detect disk degradation that is too subtle to affect overall latency but still threatens reliability
   • Move upgrade operations into a more automatic control plane for the Scylla clusters that analyses cluster state, such as cache hit rate
   • Increase the redundancy of the scylla-authentication cluster; the options being explored, not yet decided, are a higher replication factor or a hot-standby cluster in another region

   Lessons

   Taking a whole zone out of a three-zone cluster leaves no margin, so one additional slow node in another zone was enough to overload it. For some Scylla clusters the row cache hit rate of restarted nodes must reach a safe level before the next zone goes down, and a short zonal test beforehand gave false confidence. The small but critical authentication cluster needs to be run differently from Discord's much larger Scylla clusters, with more redundancy.

   View case
4. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case
5. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events

   PagerDuty2014sev1

   Symptoms

   On 3 June 2014, PagerDuty's notification pipeline performed poorly and some notifications were delayed after the Cassandra cluster behind it became heavily degraded. About 3% of events sent to the integration API could not be received.

   Database overloadHigh error rateHigh latency

   Trigger

   On 3 June 2014 the Cassandra background repair process started its normal run on one node of the cluster while other high load was on the cluster; on 4 June the team restarted the postponed repair by hand.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No headroomExcessive blast radiusHarmful manual actionIgnored early signal

   Remediation

   • Reduced the load on the Cassandra cluster
   • Stopped the Cassandra repair process
   • Cut communication between some Cassandra nodes to stabilise the cluster
   • On 4 June, deleted all in-flight data in the notification pipeline in a "factory reset"
   • Brought downstream systems back into a consistent state with the emptied pipeline by hand

   Action items

   • Scale the existing Cassandra nodes vertically to bigger, faster servers
   • Set up multiple Cassandra clusters and spread the workloads across them
   • Set system load thresholds at which the Cassandra clusters are scaled out horizontally ahead of need
   • Upgrade the current and new clusters to a more recent Cassandra version
   • Add more load shedding techniques to control Cassandra under high load
   • Bring more Cassandra expertise in-house

   Lessons

   PagerDuty concluded that its Cassandra cluster was underscaled and shared by too many services with different load and usage patterns, and that the scaling work should have come before the planned Cassandra version upgrade.

   View case

By root cause1