Cases1 case

1 to 1 of 1

1. IL-0327 After a blue/green upgrade of CircleCI's workflows database every query hit disk, and workflow starts lagged or failed

   CircleCI2025

   Symptoms

   On April 3, 2025, from 22:08 to 23:45 UTC, CircleCI customers saw higher latency and some failures when starting and cancelling workflows and jobs, and had delays and trouble viewing workflows in the UI. Nine minutes after the blue/green cutover of the workflows database, the team found increased latency in the workflows service and errors from jobs dropped after using up their 10 minutes of retries.

   High latencyHigh error rateDatabase overload

   Trigger

   A blue/green upgrade of the database behind the CircleCI workflows service cut over to the new database at 22:08 UTC.

   Root cause

   Faulty deployment

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   –

   Detection

   At 22:17 UTC the team identified increased latency in the workflows service and errors from jobs dropped after exhausting their 10-minute retries.

   Remediation

   • Restarted the workflows pods so all of them connected to the correct database
   • Upsized the new database and disabled non business critical operations on it
   • Scaled the workflows service down to a single pod while the statistics table was rebuilt
   • Re-enabled writes on the old database and restored it as primary

   Action items

   • Run an analysis procedure after every major version change in blue/green database deployments
   • Tested running analyze on a database under pressure, found it does not degrade performance further, and noted this for future remediation
   • Add automated tests and manual checkpoints throughout migrations to catch problems before the blue/green cutover

   View case

By root cause1