Incident Library

IL-0327 CircleCI · 2025-04-03database

After a blue/green upgrade of CircleCI's workflows database every query hit disk, and workflow starts lagged or failed

Around 22:00 UTC CircleCI started an upgrade of the database for the service that runs workflows, using a blue/green deployment. The analyze operation that rebuilds the database's statistics table, which the database uses for its indexes, had been run too early in the procedure, and a second major version upgrade within the same deployment made those statistics stale. After the cutover at 22:08, every query on the new database went to disk, the workflows service slowed, and jobs whose database calls exhausted their 10-minute retries were dropped. Upsizing the database, disabling non business critical operations and scaling the workflows service to one pod while the statistics were rebuilt did not help, so at 23:19 the team re-enabled writes on the old database and restored it as primary, which was complete at 23:29. The workflows queue was back to normal levels at 23:45 with no further errors. The root cause was found in the investigation after the incident.

Severity
–
Scope
Starting and cancelling workflows and jobs had increased latency and some failures, and workflows were slow or hard to view in the UI, from 22:08 to 23:45 UTC
Users affected
CircleCI customers
Region
–
Time to detect
9m
Time to resolve
1h 37m

Symptoms

On April 3, 2025, from 22:08 to 23:45 UTC, CircleCI customers saw higher latency and some failures when starting and cancelling workflows and jobs, and had delays and trouble viewing workflows in the UI. Nine minutes after the blue/green cutover of the workflows database, the team found increased latency in the workflows service and errors from jobs dropped after using up their 10 minutes of retries.

The team had expected a delay of less than 90 seconds during the database upgrade, with database calls from the workflows service sent to a queue and retried over 10 minutes, and expected the queues to grow a little. At 22:08 the blue/green deployment was complete and queries were verified as being served. At 22:30 the team restarted the workflows pods to make sure all of them were connected to the correct database, and a public incident was declared at 22:35.

At 22:41 the team saw that all queries on the new database were hitting disk, a sign that the database statistics tables had not been updated. Upsizing the database, disabling all non business critical operations on it and, at 23:00, scaling the workflows service down to a single pod to let the database recover while the statistics table was rebuilt did not improve database performance by 23:10; the workflows queue only backed up, as expected with fewer pods.

Symptom High latencyHigh error rateDatabase overload

Root cause

Root cause Faulty deployment

Component DatabaseQuery planner and table statistics
Contributing factor –
Trigger
A blue/green upgrade of the database behind the CircleCI workflows service cut over to the new database at 22:08 UTC.

Detection

At 22:17 UTC the team identified increased latency in the workflows service and errors from jobs dropped after exhausting their 10-minute retries.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Query planner and table statistics among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.