IL-0552 CircleCI · 2025-01-21database
CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UI
CircleCI's workflows service calls the archival service synchronously when it archives workflows. A code change to a function in the archival service's database on 16 January 2025 unintentionally made the database write an excessive number of log messages, and the database filled far faster than CircleCI had forecast. At about 23:00 UTC on 21 January an alert warned that the database was almost out of free storage. Halting a blue/green deployment at 23:09 to free a logical replication slot did not help. When the database reached capacity, calls from the workflows service timed out, the workflows queue grew from 23:26, and customers saw delays from 23:50. Another team scaled up the event consumer that processes the queue at 23:44, and the team on the database promoted a read replica to a standalone primary at 23:55. The workflows queue was back to normal by 00:03.
At about the same time, Linux machine jobs began to queue because provisioning instances with CircleCI's cloud provider returned errors; the provider was investigating increased API error rates on the provisioning endpoint in that region. Requests began to be fulfilled around 00:32, but the volume of requests met rate limiting, which lengthened the impact. Queues were normal at 00:56 UTC and the incident was resolved at 01:26 UTC. After the incident the database function was changed to stop the excessive logging.
- Severity
- –
- Scope
- Starting and cancelling workflows and jobs on CircleCI was slow and the CircleCI UI had trouble showing workflows; Linux machine jobs then queued (Pipelines & Workflows and CircleCI UI)
- Users affected
- CircleCI customers
- Region
- –
- Time to detect
- –
- Time to resolve
- 1h 6m
Symptoms
At about 23:00 UTC on 21 January 2025, an automated alert reported that the CircleCI database instance holding archived data was almost out of free storage space. From 23:50 UTC customers saw slow starts and cancellations of workflows and jobs, and delays and difficulty viewing workflows in the CircleCI UI, although pipelines for a specific project could still be viewed.
The first action on the database did not help. At 23:09 the team halted a blue/green deployment on the database to free a logical replication slot, suspecting the slot was the cause, but the database did not recover.
The workflows service calls the archival service synchronously, so when the archival database reached capacity those calls began timing out and the workflows queue grew from 23:26. The growth was not linked to the archival database at first, partly because a separate alert about request volume fired at about the same time. When the queue kept growing after that other issue resolved, a second team investigated workflows. After the queue recovered at 00:03, Linux machine jobs queued because provisioning instances with the cloud provider returned errors, and the volume of requests then ran into rate limiting until queues were normal at 00:56.
Symptom High latencyIntermittent timeoutsLog floodNode disk fullCloud API rate limit exhausted
Root cause
Root cause Faulty deployment
- Trigger
- A code change to a function in the archival database on 16 January 2025 unintentionally made it write an excessive number of log messages, and the database filled its storage.
Detection
An automated alert at about 23:00 UTC warned that the archival database was almost out of free storage, before customers were affected. The growing workflows queue was not linked to it at first because a separate alert about request volume fired at about the same time.
Remediation
- Scaled up the event consumer that processes the workflows queue at 23:44 UTC
- Promoted a read replica of the archival database to a standalone primary at 23:55 UTC
Action items
- Fix the database function so it no longer writes an excessive number of log messages
- Add a max duration to the workflows retry policy for archiving so it fails earlier than the default timeout
- Move workflow archival to an event-based model that decouples it from the workflows service
- Add forecast and anomaly monitoring for databases that alerts on unusual activity before it reaches critical levels
Lessons
The storage alert on the archival database did fire, but the database filled much faster than forecast, so there was little time to act before customers were affected. Because the workflows service called the archival service synchronously, a problem in the archival database slowed the whole workflows service.
Sources
- https://status.circleci.com/incidents/ym5vldzgvxv5 · official-status · archived copy
Related cases
IL-0356Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026same root cause · shared Database, CI system
IL-0324Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025same root cause · shared Database, CI system
IL-0136A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026same root cause · shared Database
IL-0570Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minuteCrowdWorks · 2026same root cause · shared Database
IL-0234An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clustersIONOS Cloud · 2026same root cause · shared CI system
IL-0510MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hoursMIT Open Learning · 2026same root cause · shared Database
IL-0380A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeksInfluxData · 2026same root cause · shared Database
IL-0369Railway's backend API went down when a column migration waited on locks and replicas could not get database connectionsRailway · 2025same root cause · shared Database
IL-0547Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an indexBuildkite · 2025same root cause · shared Database
IL-0315Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connectionRailway · 2025same root cause · shared Database
IL-0327After a blue/green upgrade of CircleCI's workflows database every query hit disk, and workflow starts lagged or failedCircleCI · 2025same root cause · shared Database
IL-0307Supabase's Postgres connection pooler Supavisor 2.0 dropped connections for about 80% of us-east-1 tenantsSupabase · 2025same root cause · shared Database
IL-0554A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutesSentry · 2024same root cause · shared Database
IL-0571STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutesSTORES · 2024same root cause · shared Database
IL-0293Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet lossGoogle Cloud · 2023same root cause · shared Database
IL-0523TANF-app production deployments were blocked for two months when a search-index migration ran out of database disk spaceU.S. Department of Health and Human Services · 2023same root cause · shared Database
IL-0567Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not playDevsisters · 2023same root cause · shared Database
IL-0483Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran outReplicate · 2023same root cause · shared Database
IL-0371Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ranRender · 2023same root cause · shared Database
IL-0524An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hourOpen Collective · 2023same root cause · shared Database
IL-0328A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"CircleCI · 2021same root cause · shared Database
IL-0566Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenanceDevsisters · 2021same root cause · shared Database
IL-0467Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hoursKubernetes project · 2019same root cause · shared CI system
IL-0466Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own servicesKubernetes project · 2019same root cause · shared CI system
IL-0391GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lockGoCardless · 2015same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0552
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0552/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.