Cases2 cases

1 to 2 of 2

1. IL-0317 Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429

   Clerk2026

   Symptoms

   At 8:11 AM PST (16:11 UTC) on 19 February 2026, Clerk's database performance dropped at once. Slow database queries made request handlers queue, and over 95% of traffic was answered with 429 without being handled; the small share of requests that reached the database returned 200, but extremely slowly.

   Service outageDatabase overloadHigh latency

   Trigger

   A routine automatic ANALYZE on one table sampled only NULL values in a column, and the Postgres planner switched a frequently run query to an inefficient plan.

   Root cause

   Query plan regression

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   Inefficient queryMonitoring blind spot

   Detection

   The incident was reported and escalated internally at 16:15 UTC, and the team started a group call.

   Remediation

   • Enabled a new failover that generates session tokens outside the core Session API
   • Re-ran ANALYZE manually on the affected table, which put the query back on its earlier plan
   • Disabled the failover once the database was back to healthy levels
   • Increased the statistics target on the table so the planner samples more rows
   • Refactored the query so the planner could plan it deterministically

   Action items

   • Add dedicated alerting for database query plan flips
   • Train the reliability team to enable the new session token failover and add automatic triggers for it
   • Audit all queries for others at risk of inefficient plans from unstable statistics
   • Name a dedicated communications lead during incidents
   • Post status page updates at a regular cadence even when nothing has changed
   • Make status page severity labels match the impact on customers
   • Cross-post incident updates to social channels

   Lessons

   Clerk says a query plan flip can cause sudden, severe degradation and needs to be detected directly instead of through downstream symptoms; relying on those symptoms led the team to blame a customer's traffic pattern. Customers also told Clerk that status page updates came too rarely, the first update came too late and the initial severity label did not match the impact.

   View case
2. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case

By root cause2