Cases2 cases

1 to 2 of 2

1. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case
2. IL-0314 Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restarting

   Daily2023

   Symptoms

   At 9:47 AM Eastern (14:47 UTC) on 7 February 2023, Daily's database alerts reported a performance problem under normal operational load: a high level of lock contention on a database that had been upgraded the weekend before and had run normally since Monday. Users could get websocket connection errors when joining calls, and REST API requests could time out or return 5xx errors.

   Lock contention or deadlockHigh error rateIntermittent timeouts

   Trigger

   Database maintenance the weekend before had upgraded the database to a MySQL point release with a low-level behaviour change, and on 7 February, under normal load, that change produced the deadlocks Daily saw on Aurora MySQL 2.11.0.

   Root cause

   Breaking change in an upgrade

   Component

   Database

   Contributing factor

   Unrepresentative stagingAutomation amplificationRetry stormChange with a delayed effect

   Detection

   Database alerts at 9:47 AM Eastern reported a performance issue with high lock contention on the newly upgraded database.

   Remediation

   • Disabled MySQL binlog replication to the old database and restarted the database to reduce load
   • The database restarted by itself at 10:47 AM, the SFUs dropped the stale meeting updates without restarting, and the platform recovered
   • Rate-limited the POST /rooms endpoint to the same limit as DELETE /rooms/:name
   • Updated an index on the affected table in production on the Saturday evening to work around the MySQL behaviour change

   Action items

   • Work with AWS so the Aurora MySQL 2.11.0 deadlock issue is documented, understood and fixed in a future release
   • Add more API request patterns to the testing workload used for infrastructure changes
   • Add production monitoring alarms on more fine-grained database metrics

   Lessons

   Daily tests infrastructure changes under production-like workloads, but its synthetic workload did not have the shape that triggers these deadlocks. A deadlock behaviour change in MySQL 5.7 that was already known turned out far more severe in production than either Daily or the AWS Aurora team expected.

   View case

By root cause1