Cases8 cases

1 to 5 of 8

1. IL-0553 CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes

   CircleCI2026

   Symptoms

   On 6 October 2026 CircleCI pipelines were slow to be created and workflows and jobs were slow to start, in short bursts from 13:30 UTC and continuously from 15:30 UTC. From 15:48 to 16:38 UTC no workflows or jobs started at all, and pipelines triggered from 16:02 UTC failed.

   High latencyService outageThroughput degradationNode disk full

   Trigger

   Around 12:40 UTC, on top of the regular workload, anti-wraparound autovacuum runs on several of the largest tables of the orchestration service's database began writing transaction logs faster than the database could archive them.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuumCI system

   Contributing factor

   Ignored early signalNo alert on limitSingle point of failure

   Detection

   Monitoring raised short alerts about errors in the orchestration service from 10:02 UTC, each clearing on its own. At 13:35 the alerts came back together with related alerts from several other services, and after they returned around 14:30 and engineers investigated further, an incident was declared at 14:54 UTC.

   Remediation

   • Changed the anti-wraparound autovacuum settings to help it finish faster
   • Moved the transaction logs onto the database's main storage volume, which took the database offline from 15:47 to 16:38 UTC
   • Changed database parameters to stop further maintenance runs from starting
   • Changed a database setting at 18:19 UTC to cut the time each write spent waiting on storage
   • Added database and server capacity to the systems that hand jobs to the execution fleet to clear the backlog faster

   Action items

   • Alert on how full the transaction log volume is and on how far archiving has fallen behind
   • Alert on the database maintenance that drives transaction log growth
   • Review all alerts so that noisy or sensitive alerts do not hide real issues
   • Tune how often and how aggressively autovacuum runs on the database
   • Track the size of the largest tables over time so they stay within safe limits
   • Remove the single transaction log bottleneck by splitting the service across several database instances or moving it to a distributed database
   • Make CircleCI's systems wait for the orchestration service to recover instead of failing jobs after retrying
   • Include specific impact and timing sooner in status page updates during long incidents

   Lessons

   CircleCI says the combination of alerts should have been recognised sooner: brief error alerts that cleared on their own were treated as routine, and nothing alerted on the transaction log volume or on archiving lag. Because every write of the orchestration service went through one database and one transaction log, a backlog in that log slowed every pipeline, workflow and job.

   View case
2. IL-0510 MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hours

   MIT Open Learning2026

   Symptoms

   On 24 March 2026 learn.mit.edu went down shortly after a regular release. An engineer reported the outage with a linked Sentry issue, and Sentry soon showed many 504 responses from the site.

   Service outageNode disk fullDatabase overloadLock contention or deadlock

   Trigger

   A regular release deployed a query pattern change to the learning resource APIs that put an aggregation inside the main query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryUnrepresentative stagingHarmful manual action

   Detection

   An engineer reported the outage with a linked Sentry issue, and Sentry showed many 504 responses. The RDS FreeStorageSpace metric showed that the database had used up its storage.

   Remediation

   • Rolled back the release's query pattern change
   • Deleted the read replica so the database could flush its buffered WAL files
   • Terminated all active queries running longer than 10 minutes with pg_terminate_backend, after the backup finished and with guidance from AWS Support

   Action items

   • Explore enabling enhanced analytics on the production Learn database
   • Document rollback steps on the platform engineering documentation site
   • Make easy rollback part of the new release management framework
   • Alert on DiskQueueDepth for production databases without making the alert too noisy
   • Alert when API and site response times are more than 30% above normal
   • Run a gameday on an example production database outage
   • Adopt Rootly for incident management, with an automatic channel and an automatic incident transcript
   • Write an outage runbook: start a call, have the on-call engineer watch metrics until they return to baseline, and give one person the communications
   • Put the query that terminates long-running queries, and a version that finds them for selective termination, into a production Postgres runbook
   • Open an issue on redesigning the Learn Resource View Event model, which is only used for counts
   • Use anonymised production data generation tools to bring RC up to production data volume, on separate infrastructure for load testing
   • Change the view count to use a precomputed column

   Lessons

   The change was reviewed by senior engineers and showed no problems locally or on RC, but data scale and load in production are very different, and nothing in the system's health showed that adding a COUNT aggregation would make disk usage explode.

   View case
3. IL-0298 Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disks

   Wikimedia Foundation2025

   Symptoms

   At 02:58 UTC on 31 March 2025, the sessionstore service in both Wikimedia data centres began returning 500 errors and edits failed. MediaWikiEditFailures (session loss) fired as critical at 03:03, and SessionStoreErrorRateHigh paged at 03:04.

   High error rateService outageNode disk fullData loss

   Trigger

   A rise in session writes, made up of session overwrites at a high rate, filled the storage of the sessionstore Cassandra nodes. The review says the rollout of SUL3 correlates strongly with it.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No alert on limitUnverified assumption

   Detection

   MediaWikiEditFailures (session loss) fired as critical at 03:03 UTC, and SessionStoreErrorRateHigh, a high 5xx rate alert, paged the batphone at 03:04. Users also reported the problem in task T390512.

   Remediation

   • Wiped the sessionstore storage, removing all sessions, and restarted Cassandra

   Action items

   • Page when the disk utilisation trend becomes unreasonable
   • Increase sessionstore storage capacity
   • Add namespacing to sessionstore
   • Add observability of the sessionstore workload
   • Identify the cause of the high rate of storage growth in sessionstore Cassandra
   • Consider an alert, without paging, on the Cassandra error rate
   • Consider paging or alerting when Cassandra nodes enter a down state

   Lessons

   The storage growth before the node failures made their timing predictable, so an alert on high storage utilisation well in advance would have given a chance to act before users were affected, and an alert on Cassandra node failures would have told responders more than an HTTP error rate alert. A cluster seen as heavily over-provisioned can still grow quickly and unsustainably under an unusual workload.

   View case
4. IL-0552 CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UI

   CircleCI2025

   Symptoms

   At about 23:00 UTC on 21 January 2025, an automated alert reported that the CircleCI database instance holding archived data was almost out of free storage space. From 23:50 UTC customers saw slow starts and cancellations of workflows and jobs, and delays and difficulty viewing workflows in the CircleCI UI, although pipelines for a specific project could still be viewed.

   High latencyIntermittent timeoutsLog floodNode disk fullCloud API rate limit exhausted

   Trigger

   A code change to a function in the archival database on 16 January 2025 unintentionally made it write an excessive number of log messages, and the database filled its storage.

   Root cause

   Faulty deployment

   Component

   DatabaseCI system

   Contributing factor

   Change with a delayed effectExcessive blast radiusOverlapping failures

   Detection

   An automated alert at about 23:00 UTC warned that the archival database was almost out of free storage, before customers were affected. The growing workflows queue was not linked to it at first because a separate alert about request volume fired at about the same time.

   Remediation

   • Scaled up the event consumer that processes the workflows queue at 23:44 UTC
   • Promoted a read replica of the archival database to a standalone primary at 23:55 UTC

   Action items

   • Fix the database function so it no longer writes an excessive number of log messages
   • Add a max duration to the workflows retry policy for archiving so it fails earlier than the default timeout
   • Move workflow archival to an event-based model that decouples it from the workflows service
   • Add forecast and anomaly monitoring for databases that alerts on unusual activity before it reaches critical levels

   Lessons

   The storage alert on the archival database did fire, but the database filled much faster than forecast, so there was little time to act before customers were affected. Because the workflows service called the archival service synchronously, a problem in the archival database slowed the whole workflows service.

   View case
5. IL-0523 TANF-app production deployments were blocked for two months when a search-index migration ran out of database disk space

   U.S. Department of Health and Human Services2023

   Symptoms

   On 20 September 2023 the sprint 82 production deployment of TANF-app failed with a disk space error while it ran a large search-index migration, search_indexes 0017, against the production PostgreSQL database on cloud.gov. Production could not be deployed again until 9 November.

   Deploys failingNode disk full

   Trigger

   The sprint 82 deployment on 20 September 2023 recreated the deleted search_indexes migration 0016 and ran the large migration search_indexes 0017 against the production database.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   Overlapping failuresGap in automated testsUnrepresentative stagingNo runbook

   Detection

   The sprint 82 production deployment failed with a disk space error, and the large migration search_indexes 0017 was identified as the cause.

   Remediation

   • Increased the production database instance size and re-ran the migrations
   • Created a backup of the production database
   • Truncated all of the search_indexes tables
   • Redeployed the sprint 82 release
   • Fixed the bug that stored empty fields as blank strings

   Action items

   • Write troubleshooting steps for database problems into the technical documentation
   • Write wiki pages and operations playbooks that outline common problems and responses
   • Add a database section and the cf revisions command to the failed-deployment troubleshooting document
   • Keep a GitHub repository of changes made directly to production
   • Save terminal output from production debugging sessions
   • Create Django migration best practices to enforce in code reviews
   • Do not edit migrations that have been merged to develop; write a new migration instead
   • Rename auto-generated migrations to more descriptive names
   • Use one auto-generated migration per model change and regenerate when a pull request makes successive changes
   • Organise a discussion on database seeding and backups for testing

   Lessons

   The work was unplanned and pulled in many developers from a small team, so the team wants documentation and playbooks that let fewer people step away from current work. Large datasets and unexpected edge cases such as rows without space-fill had not been tested, and a copy of the production database with PII scrubbed would allow tests against real-world data. Deploying in-progress code to other environments and changing migrations afterwards had caused conflicts there.

   View case

By root cause2