Cases2 cases

1 to 2 of 2

1. IL-0553 CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes

   CircleCI2026

   Symptoms

   On 6 October 2026 CircleCI pipelines were slow to be created and workflows and jobs were slow to start, in short bursts from 13:30 UTC and continuously from 15:30 UTC. From 15:48 to 16:38 UTC no workflows or jobs started at all, and pipelines triggered from 16:02 UTC failed.

   High latencyService outageThroughput degradationNode disk full

   Trigger

   Around 12:40 UTC, on top of the regular workload, anti-wraparound autovacuum runs on several of the largest tables of the orchestration service's database began writing transaction logs faster than the database could archive them.

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuumCI system

   Contributing factor

   Ignored early signalNo alert on limitSingle point of failure

   Detection

   Monitoring raised short alerts about errors in the orchestration service from 10:02 UTC, each clearing on its own. At 13:35 the alerts came back together with related alerts from several other services, and after they returned around 14:30 and engineers investigated further, an incident was declared at 14:54 UTC.

   Remediation

   • Changed the anti-wraparound autovacuum settings to help it finish faster
   • Moved the transaction logs onto the database's main storage volume, which took the database offline from 15:47 to 16:38 UTC
   • Changed database parameters to stop further maintenance runs from starting
   • Changed a database setting at 18:19 UTC to cut the time each write spent waiting on storage
   • Added database and server capacity to the systems that hand jobs to the execution fleet to clear the backlog faster

   Action items

   • Alert on how full the transaction log volume is and on how far archiving has fallen behind
   • Alert on the database maintenance that drives transaction log growth
   • Review all alerts so that noisy or sensitive alerts do not hide real issues
   • Tune how often and how aggressively autovacuum runs on the database
   • Track the size of the largest tables over time so they stay within safe limits
   • Remove the single transaction log bottleneck by splitting the service across several database instances or moving it to a distributed database
   • Make CircleCI's systems wait for the orchestration service to recover instead of failing jobs after retrying
   • Include specific impact and timing sooner in status page updates during long incidents

   Lessons

   CircleCI says the combination of alerts should have been recognised sooner: brief error alerts that cleared on their own were treated as routine, and nothing alerted on the transaction log volume or on archiving lag. Because every write of the orchestration service went through one database and one transaction log, a backlog in that log slowed every pipeline, workflow and job.

   View case
2. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers

   CircleCI2021

   Symptoms

   From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

   Throughput degradationCloud API rate limit exhaustedDatabase overload

   Trigger

   The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

   Root cause

   Breaking change in an upgrade

   Component

   Message brokerDatabase

   Contributing factor

   Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures

   Detection

   Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

   Remediation

   • Blocked Linux machine jobs from entering the work queue
   • Had GCP raise the CPU quotas
   • Deleted old VMs by hand
   • Cancelled some workflows to free resources
   • Reset the VM services deployment so the VM destroy queues had consumers again
   • Scaled up the RDS database instance, at 18:20 and again at 19:15
   • Turned down VM provisioning and blocked remote-docker jobs to shed load
   • Scaled down all pods that created and destroyed VMs
   • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
   • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes

   Action items

   • Make VM provisioning more robust to changes in load so it can recover from thundering herds
   • Make VM provisioning independent so its failure does not affect Docker jobs
   • Review and improve observability of RabbitMQ and the production queues
   • Batch requests to avoid hitting GCP API rate limits
   • Improve incident response for long-running incidents

   View case

By root cause2