Cases2 cases

1 to 2 of 2

1. IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS Proxy

   Safie2026

   Symptoms

   A few minutes after the production endpoint was switched to RDS Proxy, Safie's alert monitoring system fired on errors, and the CPU use of the Aurora MySQL writer was stuck near 100%.

   Database overloadHigh error rate

   Trigger

   The production application endpoints were switched to the new RDS Proxy all at once, with MaxConnectionsPercent at 21%, MaxIdleConnectionsPercent at 10% and ConnectionBorrowTimeout at 7 seconds, the same as the application's socket timeout, after a week in staging had shown no problem.

   Root cause

   Configuration change

   Component

   Database proxy and connection poolerDatabase

   Contributing factor

   Retry stormUnrepresentative stagingNo staged rollout

   Detection

   Safie's alert monitoring system fired on errors a few minutes after the endpoint was switched to RDS Proxy.

   Remediation

   • Rolled back the ECS Fargate tasks that used RDS Proxy to their previous version

   Action items

   • Raise MaxConnectionsPercent to 62% and MaxIdleConnectionsPercent to 31%
   • Choose between direct Aurora and RDS Proxy with a feature flag evaluated per ECS task, so the switch can be made in stages and undone without a deployment
   • Move read requests first and write requests afterwards
   • Raise the share in steps of 1%, 10%, 25%, 50%, 75% and 100%, checking proxy borrow wait, new backend connections, Aurora connections, CPU and latency and application errors before each step

   Lessons

   The failure and the later problems were not bugs in RDS Proxy; the application's connection pool implementation and settings were exposed by putting a proxy in between. With a proxy, backend connections are shared, so reuse depends on matching session state, closed transactions and connection lifetimes within the database's own timeouts. Before introducing it, the team should have checked what the application pool runs when it returns a connection, how it sets connection lifetime and what it runs on connect, more than the proxy settings themselves.

   View case
2. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case

By root cause2