IL-0553 CircleCI · 2026-10-06database
CircleCI's orchestration database filled its transaction log volume, and no workflows or jobs started for 50 minutes
CircleCI's workflow orchestration service tracks the state of every pipeline, workflow and job, decides when each job is ready, hands ready jobs to the execution fleet, and drives the job data in the UI, notifications and commit statuses. The service keeps its state in one database. The database writes every change to a transaction log on a dedicated storage volume and archives the logs continuously so the space can be reused; if archiving falls behind and the volume fills, the database stops accepting writes. On a very large table, the periodic autovacuum maintenance reads every page and writes a large volume of transaction logs.
On 6 October 2026, from around 12:40 UTC, anti-wraparound autovacuum on several of the largest tables wrote transaction logs faster than they could be archived, and the log volume began to fill. The orchestration service slowed from 13:30, short error alerts that had appeared since 10:02 were handled as routine, and the incident was declared at 14:54. Engineers tuned the autovacuum settings so it would finish faster. At 15:43 the log volume was close to full, and to keep the database from going read-only engineers moved the transaction logs to the database's main storage volume, which had plenty of free space. The move took the database offline from 15:47 for a time nobody could predict; from 15:48 to 16:38 the service processed no work, and pipelines triggered from 16:02 failed. A replacement database was prepared as a fallback, but the original came back first and stayed in service.
With logs and data on the same storage, the database ran at about half its normal rate from 16:38 to 18:19, and it had to finish archiving its log backlog before the logs could return to a dedicated volume. Engineers changed database parameters to stop more maintenance runs from starting, prepared ways to cut the work reaching the service, and prepared a complete second stack of the service that they could move to at the risk of data loss; CircleCI opted against that move. At 18:19 a database setting change cut the time each write waited on storage, processing speed recovered, and added database and server capacity for the systems that hand jobs to the execution fleet helped clear the backlog. New jobs started on time again by about 19:55, Docker jobs on larger resource classes then waited while capacity scaled up, and start times were normal by 20:30. The incident was resolved at 21:19 UTC. About 3,500 jobs failed to start, some because CircleCI's systems gave up after retrying, and CircleCI found no evidence that any job ran more than once.
- Severity
- –
- Scope
- Creating pipelines and starting workflows and jobs on CircleCI was delayed from 15:30 to 20:30 UTC and stopped from 15:48 to 16:38 UTC; pipelines triggered from 16:02 UTC failed; UI data, notifications and commit status updates were delayed; about 3,500 jobs failed to start
- Users affected
- CircleCI customers
- Region
- –
- Time to detect
- 5m
- Time to resolve
- 7h
Symptoms
On 6 October 2026 CircleCI pipelines were slow to be created and workflows and jobs were slow to start, in short bursts from 13:30 UTC and continuously from 15:30 UTC. From 15:48 to 16:38 UTC no workflows or jobs started at all, and pipelines triggered from 16:02 UTC failed.
From 10:02 UTC, monitoring raised short-lived alerts about errors in the workflow orchestration service, and each cleared on its own within about 15 minutes. Engineers had seen similar brief alerts before and the system showed no other signs of stress, so each time the engineers followed the runbook and also began work to make the alerts less sensitive. At 13:35 the alerts returned together with related alerts from several other services, and again around 14:30, and an incident was declared at 14:54. No alert existed on how full the transaction log volume was or on how far log archiving had fallen behind.
Once the slowdown was traced to an anti-wraparound autovacuum, cancelling it was no answer, because the database restarts that process automatically when it is cancelled. Engineers changed its settings to help it finish faster, but by 15:30 the delays were continuous and some jobs began to fail.
UI data, notifications and commit status updates to version control providers were delayed as well; early status updates said job, workflow and pipeline data could take up to 15 minutes to appear in the UI and outbound notifications and webhooks were up to 1 minute late. After the database came back at 16:38 it processed work at about half its normal rate until 18:19, and jobs waited tens of minutes to start, up to about 50 minutes at the longest. When the backlog reached the execution fleet, some Docker jobs on larger resource classes waited up to about 10 minutes from 19:49 to 20:30 while capacity scaled up; the status page put the peak at around 20:05 UTC, with waits of up to about 8 minutes for 2xlarge gen2 and 6 minutes for xlarge gen2. About 3,500 jobs failed to start.
Symptom High latencyService outageThroughput degradationNode disk full
Root cause
Root cause Capacity exhaustion
- Trigger
- Around 12:40 UTC, on top of the regular workload, anti-wraparound autovacuum runs on several of the largest tables of the orchestration service's database began writing transaction logs faster than the database could archive them.
Detection
Monitoring raised short alerts about errors in the orchestration service from 10:02 UTC, each clearing on its own. At 13:35 the alerts came back together with related alerts from several other services, and after they returned around 14:30 and engineers investigated further, an incident was declared at 14:54 UTC.
Remediation
- Changed the anti-wraparound autovacuum settings to help it finish faster
- Moved the transaction logs onto the database's main storage volume, which took the database offline from 15:47 to 16:38 UTC
- Changed database parameters to stop further maintenance runs from starting
- Changed a database setting at 18:19 UTC to cut the time each write spent waiting on storage
- Added database and server capacity to the systems that hand jobs to the execution fleet to clear the backlog faster
Action items
- Alert on how full the transaction log volume is and on how far archiving has fallen behind
- Alert on the database maintenance that drives transaction log growth
- Review all alerts so that noisy or sensitive alerts do not hide real issues
- Tune how often and how aggressively autovacuum runs on the database
- Track the size of the largest tables over time so they stay within safe limits
- Remove the single transaction log bottleneck by splitting the service across several database instances or moving it to a distributed database
- Make CircleCI's systems wait for the orchestration service to recover instead of failing jobs after retrying
- Include specific impact and timing sooner in status page updates during long incidents
Lessons
CircleCI says the combination of alerts should have been recognised sooner: brief error alerts that cleared on their own were treated as routine, and nothing alerted on the transaction log volume or on archiving lag. Because every write of the orchestration service went through one database and one transaction log, a backlog in that log slowed every pipeline, workflow and job.
Sources
- https://status.circleci.com/incidents/z4hbq8h49j8c · official-status · archived copy
Related cases
IL-0550Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pendingAtlassian · 2024same root cause · shared Database, PostgreSQL vacuum
IL-0456Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQLInflab · 2022same root cause · shared Database, PostgreSQL vacuum
IL-0211A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hoursMailchimp · 2019same root cause · shared Database, PostgreSQL vacuum
IL-0450Oracle RAC commits behind order APIs kept waiting over 0.1 s on log file sync while average host CPU had headroomToss Securities · 2026same root cause · shared Database
IL-0379InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026same root cause · shared Database
IL-0471GitHub Actions runs failed or sat queued for 9 hours after a routine deployment's pod replacement saturated capacityGitHub · 2026same root cause · shared CI system
IL-0309Preset's RDS PostgreSQL 14.7 to 17.9 upgrade in US East 1 ran out of storage and took Superset down for six hoursPreset · 2026same root cause · shared Database
IL-0435Error rates rose as Aurora MySQL killed connections and declined queries to avoid OOM while load metrics stayed flatCyberAgent · 2026same root cause · shared Database
IL-0497Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passedButtondown · 2026same root cause · shared Database
IL-0288ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026same root cause · shared Database
IL-0350Hosts in an IONOS TXL cluster lost connectivity when an InfiniBand fabric resweep failed on a full multicast tableIONOS Cloud · 2026same root cause · shared Database
IL-0305Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026same root cause · shared Database
IL-0390PostHog US Cloud event ingestion slowed to delays of up to 2 days as writes to the Persons Postgres table stalledPostHog · 2025same root cause · shared Database
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Database
IL-0287One overloaded shard of MongoDB's internal monitoring database delayed Atlas metrics, alerts and cluster operationsMongoDB · 2025same root cause · shared Database
IL-0557Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hoursStorj · 2025same root cause · shared Database
IL-0551Bitbucket Cloud's overloaded primary database failed over, and the website, API and Pipelines were slow for almost a dayAtlassian · 2025same root cause · shared Database
IL-0298Wikimedia edits failed for 38 minutes when sessionstore Cassandra nodes in both data centres crashed on full disksWikimedia Foundation · 2025same root cause · shared Database
IL-0312Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went downGadget · 2024same root cause · shared Database
IL-0544Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024same root cause · shared Database
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Database
IL-0375MongoDB Atlas cluster changes and alerts lagged for hours after a control plane metadata migration congested a databaseMongoDB · 2024same root cause · shared Database
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Database
IL-0270Discord's ScyllaDB authentication cluster overloaded during an OS upgrade, and logins failed for about 50 minutesDiscord · 2023same root cause · shared Database
IL-0202A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recoverHoneycomb · 2023 · sev1same root cause · shared Database
IL-0386After a move to Google Cloud, Postgres connection timeouts and memcached i/o timeouts spiked on GKE nodesincident.io · 2023same root cause · shared Database
IL-0326A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hoursMailPace · 2023same root cause · shared Database
IL-0546Buildkite jobs could not go to newly connected agents once agent IDs passed a 32-bit integer foreign key's limitBuildkite · 2023same root cause · shared Database
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022same root cause · shared Database
IL-0301Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice37signals · 2022same root cause · shared Database
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Database
IL-0302Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022same root cause · shared Database
IL-0212Many Slack users could not connect because a Vitess keyspace for channel membership was overloaded by cache-miss queriesSlack · 2022same root cause · shared Database
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Database
IL-0496Ably's Cassandra hit 100% CPU in every region and queries timed out globally, and the load returned on two more eveningsAbly · 2020same root cause · shared Database
IL-0443A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019same root cause · shared Database
IL-0002CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared CI system
IL-0401Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panicsKubernetes project · 2019same root cause · shared CI system
IL-0037Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019same root cause · shared Database
IL-0297Mezmo users could not sign in and log indexing lagged after the auth cache's Redis servers were OOM killedMezmo · 2019same root cause · shared Database
IL-0064CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Database
IL-0300Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,64737signals · 2018same root cause · shared Database
IL-0334Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017same root cause · shared Database
IL-0333A downsized RDS PostgreSQL database maxed out its CPU at US peak and failing ELB health checks took buildkite.com downBuildkite · 2016same root cause · shared Database
IL-0282DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata serviceAmazon Web Services · 2015same root cause · shared Database
IL-0383PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming eventsPagerDuty · 2014 · sev1same root cause · shared Database
IL-0320Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0553
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0553/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.