Cases2 cases

1 to 2 of 2

1. IL-0318 Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk

   Clerk2026

   Symptoms

   At 15:57 UTC on 10 March 2026, Clerk's alerting reported increased latency and elevated 5xx errors across its APIs, and the incident was declared. Monitoring showed increased lock contention in the database, which had started about a minute earlier.

   High latencyHigh error rateLock contention or deadlockService outage

   Trigger

   A routine live migration of the Google Cloud SQL virtual machine that hosts Clerk's Postgres database, done without advance notice to Clerk, did not work as intended.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Limited debug access

   Detection

   Alerting reported increased latency and elevated 5xx errors across the APIs at 15:57 UTC, and monitoring showed lock contention in the database that had begun about a minute earlier.

   Remediation

   • Enabled Origin Outage Mode to keep session management working
   • Opened a ticket with GCP after seeing elevated IO wait on reads
   • Disabled Origin Outage Mode after the live migration completed and the system stabilised

   Action items

   • Ask Google to pin the database again to stop further live migrations
   • Use replica promotion instead of live migrations for all Postgres database maintenance
   • Move to another database provider or run Postgres in-house unless Google offers a way to disable live migrations permanently
   • Return 500 instead of the internal service's 429 so applications can tell the state of Clerk's systems from the error code
   • Shift most of the engineering team to reliability-focused projects

   Lessons

   Clerk concludes that Cloud SQL live migrations are still unsafe after months of joint work with Google, and that they are opaque, leaving Clerk to trust Google's word. Clerk prefers traditional replica promotion for all Postgres maintenance, an automated process it had already used without incident for a Postgres major version upgrade on January 17, 2026.

   View case
2. IL-0317 Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429

   Clerk2026

   Symptoms

   At 8:11 AM PST (16:11 UTC) on 19 February 2026, Clerk's database performance dropped at once. Slow database queries made request handlers queue, and over 95% of traffic was answered with 429 without being handled; the small share of requests that reached the database returned 200, but extremely slowly.

   Service outageDatabase overloadHigh latency

   Trigger

   A routine automatic ANALYZE on one table sampled only NULL values in a column, and the Postgres planner switched a frequently run query to an inefficient plan.

   Root cause

   Query plan regression

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   Inefficient queryMonitoring blind spot

   Detection

   The incident was reported and escalated internally at 16:15 UTC, and the team started a group call.

   Remediation

   • Enabled a new failover that generates session tokens outside the core Session API
   • Re-ran ANALYZE manually on the affected table, which put the query back on its earlier plan
   • Disabled the failover once the database was back to healthy levels
   • Increased the statistics target on the table so the planner samples more rows
   • Refactored the query so the planner could plan it deterministically

   Action items

   • Add dedicated alerting for database query plan flips
   • Train the reliability team to enable the new session token failover and add automatic triggers for it
   • Audit all queries for others at risk of inefficient plans from unstable statistics
   • Name a dedicated communications lead during incidents
   • Post status page updates at a regular cadence even when nothing has changed
   • Make status page severity labels match the impact on customers
   • Cross-post incident updates to social channels

   Lessons

   Clerk says a query plan flip can cause sudden, severe degradation and needs to be detected directly instead of through downstream symptoms; relying on those symptoms led the team to blame a customer's traffic pattern. Customers also told Clerk that status page updates came too rarely, the first update came too late and the initial severity label did not match the impact.

   View case

By root cause2