Cases1 case

1 to 1 of 1

1. IL-0318 Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk

   Clerk2026

   Symptoms

   At 15:57 UTC on 10 March 2026, Clerk's alerting reported increased latency and elevated 5xx errors across its APIs, and the incident was declared. Monitoring showed increased lock contention in the database, which had started about a minute earlier.

   High latencyHigh error rateLock contention or deadlockService outage

   Trigger

   A routine live migration of the Google Cloud SQL virtual machine that hosts Clerk's Postgres database, done without advance notice to Clerk, did not work as intended.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Limited debug access

   Detection

   Alerting reported increased latency and elevated 5xx errors across the APIs at 15:57 UTC, and monitoring showed lock contention in the database that had begun about a minute earlier.

   Remediation

   • Enabled Origin Outage Mode to keep session management working
   • Opened a ticket with GCP after seeing elevated IO wait on reads
   • Disabled Origin Outage Mode after the live migration completed and the system stabilised

   Action items

   • Ask Google to pin the database again to stop further live migrations
   • Use replica promotion instead of live migrations for all Postgres database maintenance
   • Move to another database provider or run Postgres in-house unless Google offers a way to disable live migrations permanently
   • Return 500 instead of the internal service's 429 so applications can tell the state of Clerk's systems from the error code
   • Shift most of the engineering team to reliability-focused projects

   Lessons

   Clerk concludes that Cloud SQL live migrations are still unsafe after months of joint work with Google, and that they are opaque, leaving Clerk to trust Google's word. Clerk prefers traditional replica promotion for all Postgres maintenance, an automated process it had already used without incident for a Postgres major version upgrade on January 17, 2026.

   View case

By root cause1