Incident Library

IL-0318 Clerk · 2026-03-10databasegcp

Clerk's APIs went down with lock contention and 429s while a Google Cloud SQL live migration slowed its database disk

Clerk runs Postgres on Google Cloud SQL Enterprise Plus with high availability. Cloud SQL performs live migrations of its virtual machines on a routine basis, without telling Clerk in advance, and when they work they do not affect availability. On 10 March 2026 a live migration did not work as expected: disk latency on Clerk's database rose sharply, lock contention followed, and the service went down completely. Clerk enabled Origin Outage Mode at 16:10 UTC to keep session management working. At 16:23 the problem ended on its own when the migration completed, Clerk turned Origin Outage Mode off shortly after, and GCP later told Clerk the cause.

This was not the first such failure. In September 2025 a different live migration had caused a major Clerk incident, from which Clerk learned that a failed live migration causes downtime until it completes and that a replica failover during one is unsafe. Google escalated that ticket to P0 and pinned the database to its datacenter so no further live migrations would run. Over the following months, with weekly calls and over 80 Google staff involved, Clerk worked, at Google's guidance, on reducing its average queries per request, introducing Google's managed connection pooler and increasing the Postgres version, while Google looked for the issue in its live migration process. The database was unpinned in January, and the first live migration after that succeeded with both teams on a call.

Severity
–
Scope
Clerk's APIs: increased latency, 5xx errors and then 429 responses, and a complete service outage; session management kept working from 16:10 UTC through Origin Outage Mode
Users affected
Clerk customers and the users of their applications
Region
–
Time to detect
1m
Time to resolve
27m

Symptoms

At 15:57 UTC on 10 March 2026, Clerk's alerting reported increased latency and elevated 5xx errors across its APIs, and the incident was declared. Monitoring showed increased lock contention in the database, which had started about a minute earlier.

The database stayed up with CPU usage within normal limits, but queries and transactions took far longer to complete, which saturated Clerk's compute resources, and incoming requests began returning 429. Session outage resiliency triggered automatically at origin to keep serving session token requests, but because compute was already saturated most of those requests got 429 as well. The 429 came straight from an internal service instead of being turned into a 500.

There was no clear cause at first. Moving all reads to the replicas to relieve the primary did not help. During the investigation the team saw elevated IO wait on reads without a matching increase in writes, suspected an infrastructure problem and opened a ticket with GCP.

Symptom High latencyHigh error rateLock contention or deadlockService outage

Root cause

Root cause Cloud provider outage

Component Database
Contributing factor Limited debug access
Trigger
A routine live migration of the Google Cloud SQL virtual machine that hosts Clerk's Postgres database, done without advance notice to Clerk, did not work as intended.

Detection

Alerting reported increased latency and elevated 5xx errors across the APIs at 15:57 UTC, and monitoring showed lock contention in the database that had begun about a minute earlier.

Remediation

Action items

Lessons

Clerk concludes that Cloud SQL live migrations are still unsafe after months of joint work with Google, and that they are opaque, leaving Clerk to trust Google's word. Clerk prefers traditional replica promotion for all Postgres maintenance, an automated process it had already used without incident for a Postgres major version upgrade on January 17, 2026.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.