Cases3 cases

1 to 3 of 3

1. IL-0317 Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429

   Clerk2026

   Symptoms

   At 8:11 AM PST (16:11 UTC) on 19 February 2026, Clerk's database performance dropped at once. Slow database queries made request handlers queue, and over 95% of traffic was answered with 429 without being handled; the small share of requests that reached the database returned 200, but extremely slowly.

   Service outageDatabase overloadHigh latency

   Trigger

   A routine automatic ANALYZE on one table sampled only NULL values in a column, and the Postgres planner switched a frequently run query to an inefficient plan.

   Root cause

   Query plan regression

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   Inefficient queryMonitoring blind spot

   Detection

   The incident was reported and escalated internally at 16:15 UTC, and the team started a group call.

   Remediation

   • Enabled a new failover that generates session tokens outside the core Session API
   • Re-ran ANALYZE manually on the affected table, which put the query back on its earlier plan
   • Disabled the failover once the database was back to healthy levels
   • Increased the statistics target on the table so the planner samples more rows
   • Refactored the query so the planner could plan it deterministically

   Action items

   • Add dedicated alerting for database query plan flips
   • Train the reliability team to enable the new session token failover and add automatic triggers for it
   • Audit all queries for others at risk of inefficient plans from unstable statistics
   • Name a dedicated communications lead during incidents
   • Post status page updates at a regular cadence even when nothing has changed
   • Make status page severity labels match the impact on customers
   • Cross-post incident updates to social channels

   Lessons

   Clerk says a query plan flip can cause sudden, severe degradation and needs to be detected directly instead of through downstream symptoms; relying on those symptoms led the team to blame a customer's traffic pattern. Customers also told Clerk that status page updates came too rarely, the first update came too late and the initial severity label did not match the impact.

   View case
2. IL-0327 After a blue/green upgrade of CircleCI's workflows database every query hit disk, and workflow starts lagged or failed

   CircleCI2025

   Symptoms

   On April 3, 2025, from 22:08 to 23:45 UTC, CircleCI customers saw higher latency and some failures when starting and cancelling workflows and jobs, and had delays and trouble viewing workflows in the UI. Nine minutes after the blue/green cutover of the workflows database, the team found increased latency in the workflows service and errors from jobs dropped after using up their 10 minutes of retries.

   High latencyHigh error rateDatabase overload

   Trigger

   A blue/green upgrade of the database behind the CircleCI workflows service cut over to the new database at 22:08 UTC.

   Root cause

   Faulty deployment

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   –

   Detection

   At 22:17 UTC the team identified increased latency in the workflows service and errors from jobs dropped after exhausting their 10-minute retries.

   Remediation

   • Restarted the workflows pods so all of them connected to the correct database
   • Upsized the new database and disabled non business critical operations on it
   • Scaled the workflows service down to a single pod while the statistics table was rebuilt
   • Re-enabled writes on the old database and restored it as primary

   Action items

   • Run an analysis procedure after every major version change in blue/green database deployments
   • Tested running analyze on a database under pressure, found it does not degrade performance further, and noted this for future remediation
   • Add automated tests and manual checkpoints throughout migrations to catch problems before the blue/green cutover

   View case
3. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case

By root cause3