Cases2 cases

1 to 2 of 2

1. IL-0318 Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk

   Clerk2026

   Symptoms

   At 15:57 UTC on 10 March 2026, Clerk's alerting reported increased latency and elevated 5xx errors across its APIs, and the incident was declared. Monitoring showed increased lock contention in the database, which had started about a minute earlier.

   High latencyHigh error rateLock contention or deadlockService outage

   Trigger

   A routine live migration of the Google Cloud SQL virtual machine that hosts Clerk's Postgres database, done without advance notice to Clerk, did not work as intended.

   Root cause

   Cloud provider outage

   Component

   Database

   Contributing factor

   Limited debug access

   Detection

   Alerting reported increased latency and elevated 5xx errors across the APIs at 15:57 UTC, and monitoring showed lock contention in the database that had begun about a minute earlier.

   Remediation

   • Enabled Origin Outage Mode to keep session management working
   • Opened a ticket with GCP after seeing elevated IO wait on reads
   • Disabled Origin Outage Mode after the live migration completed and the system stabilised

   Action items

   • Ask Google to pin the database again to stop further live migrations
   • Use replica promotion instead of live migrations for all Postgres database maintenance
   • Move to another database provider or run Postgres in-house unless Google offers a way to disable live migrations permanently
   • Return 500 instead of the internal service's 429 so applications can tell the state of Clerk's systems from the error code
   • Shift most of the engineering team to reliability-focused projects

   Lessons

   Clerk concludes that Cloud SQL live migrations are still unsafe after months of joint work with Google, and that they are opaque, leaving Clerk to trust Google's word. Clerk prefers traditional replica promotion for all Postgres maintenance, an automated process it had already used without incident for a Postgres major version upgrade on January 17, 2026.

   View case
2. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case

By root cause2