Incident Library
Applied filters ComponentDatabaseSymptomHigh error rateFactorLimited debug accessClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0318 Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk
    Clerk2026
    Symptoms

    At 15:57 UTC on 10 March 2026, Clerk's alerting reported increased latency and elevated 5xx errors across its APIs, and the incident was declared. Monitoring showed increased lock contention in the database, which had started about a minute earlier.

    High latencyHigh error rateLock contention or deadlockService outage
    Trigger

    A routine live migration of the Google Cloud SQL virtual machine that hosts Clerk's Postgres database, done without advance notice to Clerk, did not work as intended.

    Root cause
    Cloud provider outage
    Component
    Database
    Contributing factor
    Limited debug access
    Detection

    Alerting reported increased latency and elevated 5xx errors across the APIs at 15:57 UTC, and monitoring showed lock contention in the database that had begun about a minute earlier.

    Remediation
    • Enabled Origin Outage Mode to keep session management working
    • Opened a ticket with GCP after seeing elevated IO wait on reads
    • Disabled Origin Outage Mode after the live migration completed and the system stabilised
    Action items
    • Ask Google to pin the database again to stop further live migrations
    • Use replica promotion instead of live migrations for all Postgres database maintenance
    • Move to another database provider or run Postgres in-house unless Google offers a way to disable live migrations permanently
    • Return 500 instead of the internal service's 429 so applications can tell the state of Clerk's systems from the error code
    • Shift most of the engineering team to reliability-focused projects
    Lessons

    Clerk concludes that Cloud SQL live migrations are still unsafe after months of joint work with Google, and that they are opaque, leaving Clerk to trust Google's word. Clerk prefers traditional replica promotion for all Postgres maintenance, an automated process it had already used without incident for a Postgres major version upgrade on January 17, 2026.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Cloud provider outage1 case

Component
Database
Contributing factor
Limited debug access
Organisation
Clerk
  1. IL-0318 Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk
    Clerk2026