Cases5 cases

1 to 5 of 5

1. IL-0553 CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes

   CircleCI2026

   Symptoms

   On 6 October 2026 CircleCI pipelines were slow to be created and workflows and jobs were slow to start, in short bursts from 13:30 UTC and continuously from 15:30 UTC. From 15:48 to 16:38 UTC no workflows or jobs started at all, and pipelines triggered from 16:02 UTC failed.

   High latencyService outageThroughput degradationNode disk full

   Trigger

   Around 12:40 UTC, on top of the regular workload, anti-wraparound autovacuum runs on several of the largest tables of the orchestration service's database began writing transaction logs faster than the database could archive them.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuumCI system

   Contributing factor

   Ignored early signalNo alert on limitSingle point of failure

   Detection

   Monitoring raised short alerts about errors in the orchestration service from 10:02 UTC, each clearing on its own. At 13:35 the alerts came back together with related alerts from several other services, and after they returned around 14:30 and engineers investigated further, an incident was declared at 14:54 UTC.

   Remediation

   • Changed the anti-wraparound autovacuum settings to help it finish faster
   • Moved the transaction logs onto the database's main storage volume, which took the database offline from 15:47 to 16:38 UTC
   • Changed database parameters to stop further maintenance runs from starting
   • Changed a database setting at 18:19 UTC to cut the time each write spent waiting on storage
   • Added database and server capacity to the systems that hand jobs to the execution fleet to clear the backlog faster

   Action items

   • Alert on how full the transaction log volume is and on how far archiving has fallen behind
   • Alert on the database maintenance that drives transaction log growth
   • Review all alerts so that noisy or sensitive alerts do not hide real issues
   • Tune how often and how aggressively autovacuum runs on the database
   • Track the size of the largest tables over time so they stay within safe limits
   • Remove the single transaction log bottleneck by splitting the service across several database instances or moving it to a distributed database
   • Make CircleCI's systems wait for the orchestration service to recover instead of failing jobs after retrying
   • Include specific impact and timing sooner in status page updates during long incidents

   Lessons

   CircleCI says the combination of alerts should have been recognised sooner: brief error alerts that cleared on their own were treated as routine, and nothing alerted on the transaction log volume or on archiving lag. Because every write of the orchestration service went through one database and one transaction log, a backlog in that log slowed every pipeline, workflow and job.

   View case
2. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks

   Wikimedia Foundation2025

   Symptoms

   At 02:58 UTC on 31 March 2025, the sessionstore service in both Wikimedia data centres began returning 500 errors and edits failed. MediaWikiEditFailures (session loss) fired as critical at 03:03, and SessionStoreErrorRateHigh paged at 03:04.

   High error rateService outageNode disk fullData loss

   Trigger

   A rise in session writes, made up of session overwrites at a high rate, filled the storage of the sessionstore Cassandra nodes. The review says the rollout of SUL3 correlates strongly with it.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No alert on limitUnverified assumption

   Detection

   MediaWikiEditFailures (session loss) fired as critical at 03:03 UTC, and SessionStoreErrorRateHigh, a high 5xx rate alert, paged the batphone at 03:04. Users also reported the problem in task T390512.

   Remediation

   • Wiped the sessionstore storage, removing all sessions, and restarted Cassandra

   Action items

   • Page when the disk utilisation trend becomes unreasonable
   • Increase sessionstore storage capacity
   • Add namespacing to sessionstore
   • Add observability of the sessionstore workload
   • Identify the cause of the high rate of storage growth in sessionstore Cassandra
   • Consider an alert, without paging, on the Cassandra error rate
   • Consider paging or alerting when Cassandra nodes enter a down state

   Lessons

   The storage growth before the node failures made their timing predictable, so an alert on high storage utilisation well in advance would have given a chance to act before users were affected, and an alert on Cassandra node failures would have told responders more than an HTTP error rate alert. A cluster seen as heavily over-provisioned can still grow quickly and unsustainably under an unusual workload.

   View case
3. IL-0326 A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours

   MailPace2023

   Symptoms

   At 13:36 BST on Friday August 11th 2023, minutes after MailPace reran its data clean-up job with more memory, notifications arrived that all database transactions were failing across multiple services. The managed Postgres database had gone offline, did not come back when restarted, and reported that it was out of space.

   Node disk fullService outageWrites rejected

   Trigger

   The daily retention clean-up job, which had been failing with killed, was rerun with its memory raised from 1gb to 8gb, and its large destroy_all deletions filled the database disk.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   Inefficient queryMonitoring blind spotNo headroom

   Detection

   Notifications that all database transactions were failing across multiple services arrived as soon as the outage began; the provider was automatically notified at 99% storage. MailPace's two status pages did not detect the outage.

   Remediation

   • Contacted the managed database provider's support
   • Notified users on Twitter and Mastodon
   • Prepared an offsite backup from six hours earlier on a new database instance as a fallback
   • The provider migrated the database to a larger instance
   • Removed the old data in production by hand, day by day, with an updated query

   Action items

   • Delete old data in batches of 50 records with in_batches in the clean-up job
   • Run the clean-up job four times a day instead of once
   • Stay on a much larger database instance with room to grow
   • Add cron monitoring to all cron jobs
   • Point the status pages at a /health endpoint that checks database access and returns a 500 error on failure
   • Investigate a hot-replica database for automated failover

   Lessons

   Several causes came together: a clean-up job that failed silently, a naive deletion that grew huge once data had piled up, a database plan without headroom for growth, and status pages that missed the outage, presumably because https://app.mailpace.com still answered 200 while the database was down. MailPace also advises that critical services such as email sending be set up with more than one provider.

   View case
4. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours

   Mailchimp2019

   Symptoms

   From 05:35 UTC on 4 February 2019 Mandrill, Mailchimp's transactional email service, sent only 80% of queued emails, because one of its PostgreSQL shards, shard4, stopped accepting writes. When Mandrill engineers opened an investigation at 11:58 UTC, they saw a sharply elevated load on shard4.

   Writes rejectedThroughput degradationLoad imbalanceNode disk full

   Trigger

   At 05:35 UTC on 4 February the transaction ID counter of shard4, the busiest Mandrill PostgreSQL shard, reached its upper limit and PostgreSQL shut the database down to protect against wraparound.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Ignored early signalNo alert on limitSignal buried in noiseRetry stormMonitoring blind spot

   Detection

   The first exception was logged overnight but was buried by unrelated exceptions in the centralized error reporting. Mandrill engineers opened an investigation at 11:58 UTC and, while looking for the source of the rising load on shard4, found the wraparound at 12:56 UTC.

   Remediation

   • Put shard4 into standalone mode and started VACUUM at 14:16 UTC on 4 February
   • Stopped and restarted the vacuum with settings that made it run faster
   • Started a dump and restore of the database without the Search and Url tables while the vacuum ran
   • Replaced storage volumes on the Mandrill app servers whose disks were filling with queued jobs and logs
   • Moved the job-queue tables from shard4 to other live shards
   • Truncated the Search and Url tables on the locked database at 19:12 UTC on 5 February, after which the vacuum finished in about an hour
   • Started sending from the backlog at 21:36 UTC
   • Added alerting for XID wraparound by 22:40 UTC

   Lessons

   Mailchimp named the factors that slowed its response: knowledge of and access to Mandrill systems and monitoring rested with a few people, Mandrill logging needed a locally writable disk, logs of malformed API requests were hard to separate from real application errors, job health was hard to see, and the disk space problem made partial deployments for triage risky. What helped was trained incident commanders with clear roles (incident commanders in 3-hour rotations, scribes, technical liaisons and points of contact between teams), periodic summaries with a list of recent and open decisions, and taking time in a video meeting to reassess, which led to the truncation. Running in the cloud let the team expand storage quickly, and the team was lucky that the largest table holding up the vacuum held data it could delete.

   View case
5. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed

   Postmark2011

   Symptoms

   At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

   Node disk fullWrites rejectedFailover failure

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

   Remediation

   • Tried to free up logs and disk space on the primary server
   • Failed over manually to the second MySQL server
   • Recovered the jobs that had failed
   • Sent the emails that each server had queued locally while the database was unavailable

   Action items

   • Upgrade the MySQL cluster with a focus on availability and performance
   • Make the alerts more sensitive to catch problems sooner

   Lessons

   Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

   View case

By root cause1