Cases2 cases

1 to 2 of 2

1. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case
2. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022

   GitHub2022

   Symptoms

   On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

   Writes rejectedDatabase overloadHigh error rateFailover failure

   Trigger

   Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius

   Remediation

   • Failed over mysql1 to a healthy replica on March 16
   • Reset the application connections to the new primary on March 17
   • Added an index that fixed the main query performance problem
   • Performed a primary failover on March 22
   • Performed a primary failover on March 23
   • Throttled webhook traffic to reduce load

   Action items

   • Keep throttling webhook traffic at peak load times
   • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
   • Move traffic to other databases to reduce load and speed up failover
   • Review change management, particularly monitoring and changes during high load in production
   • Shard the databases and scale up hardware

   View case

By root cause2