Incident Library

IL-0498 Clerk · 2025-09-14databasegcp

Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

Clerk runs its services on Cloud Run with many containers. The connection pooler in each container had a static maximum connection lifetime of 15 minutes, so each connection was closed and replaced every 15 minutes. Before the upgrade, Postgres' lock manager granted new connections with O(n²) time complexity, which acted as a natural rate limiter and spread the replacement of expired connections over time. The minor version that the cloud provider applied on 14 September 2025 optimised connection granting, so large batches of new connections could be granted almost at once.

With fast connection granting, all expired connections in a container were recreated at the same moment. During a deploy the Cloud Run containers start one after another within about a minute, which keeps their recycling apart, but any back pressure in the database gradually pulled the recycling of all containers into step. Once synchronised, every container refreshed its connections at the same time every 15 minutes, the effect grew with each cycle, the pool of active connections ran out and customers saw latency spikes. The Backend API was not affected in the same way because it manages far fewer connections. A release freeze during the incident took away the staggering that normal deploys give connection creation and made the problem worse.

From 15 to 18 September engineers optimised expensive queries, refined critical indexes and re-indexed, blocked abusive traffic more aggressively, which may have rejected a small number of legitimate requests with HTTP 429, performed a manual minor version upgrade and increased database capacity. The provider was unwilling to roll the database back to the earlier version. At 03:19 UTC on 18 September the team identified the cause and added jitter to the connection pooling configuration. Database load dropped below pre-incident levels at once and the latency spikes stopped.

Severity
–
Scope
Clerk customer traffic: intermittent significant request failures and latency spikes; for 3 to 6 minutes on 17 September new sign ins, sign ups and other account management failed, while user sessions stayed active
Users affected
–
Region
–
Time to detect
0m
Time to resolve
62h 11m

Symptoms

At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

The database had already looked different the day before. Shortly after the cloud provider automatically upgraded the database to a new minor version at 05:30 UTC on 14 September, monitoring showed an increased but still acceptable database load with no effect on latency. On 15 September latency returned to normal at 14:09 UTC, and significant failures and latency spikes came back from 14:58 to 15:16 UTC that day and from 15:07 to 15:14 UTC on 17 September. The latency spikes showed in Clerk's Frontend API but not consistently in its Backend API, which has the same connection pooling logic.

On 17 September a deploy with a new query optimisation at 00:22 UTC seemed to remove the load spikes, but they were back at 01:00 UTC and now came roughly every 15 minutes. Session management kept working through these spikes, because the session endpoints tolerate more latency and Clerk's SDKs retry. A manual minor version upgrade and an increase in database capacity from 07:15 to 08:10 UTC caused 3 to 6 minutes of failures for new sign ins, sign ups and other account management, and the spikes again seemed to be gone until the afternoon.

Several things hid the cause. Database connection metrics were sampled every 60 seconds while the connection recycling happened within seconds every 15 minutes, so dashboards showed only occasional spikes in active connections, and the team first treated them as a symptom of load. In the same days Clerk was mitigating a large volume of fraudulent sign ups whose load surges came about every 10 minutes, and recurring background jobs and scheduled queries ran on 10, 15 and 30-minute intervals, so the team first suspected the attack and those jobs. A release freeze imposed to reduce risk during the incident made the problem worse.

Symptom High error rateHigh latencyDatabase overloadConnection pool exhaustion

Root cause

Root cause Breaking change in an upgrade

Component DatabaseDatabase proxy and connection pooler
Contributing factor Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action
Trigger
At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

Detection

Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

Remediation

Action items

Lessons

Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database proxy and connection pooler among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.