Cases4 cases

1 to 4 of 4

1. IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS Proxy

   Safie2026

   Symptoms

   A few minutes after the production endpoint was switched to RDS Proxy, Safie's alert monitoring system fired on errors, and the CPU use of the Aurora MySQL writer was stuck near 100%.

   Database overloadHigh error rate

   Trigger

   The production application endpoints were switched to the new RDS Proxy all at once, with MaxConnectionsPercent at 21%, MaxIdleConnectionsPercent at 10% and ConnectionBorrowTimeout at 7 seconds, the same as the application's socket timeout, after a week in staging had shown no problem.

   Root cause

   Configuration change

   Component

   Database proxy and connection poolerDatabase

   Contributing factor

   Retry stormUnrepresentative stagingNo staged rollout

   Detection

   Safie's alert monitoring system fired on errors a few minutes after the endpoint was switched to RDS Proxy.

   Remediation

   • Rolled back the ECS Fargate tasks that used RDS Proxy to their previous version

   Action items

   • Raise MaxConnectionsPercent to 62% and MaxIdleConnectionsPercent to 31%
   • Choose between direct Aurora and RDS Proxy with a feature flag evaluated per ECS task, so the switch can be made in stages and undone without a deployment
   • Move read requests first and write requests afterwards
   • Raise the share in steps of 1%, 10%, 25%, 50%, 75% and 100%, checking proxy borrow wait, new backend connections, Aurora connections, CPU and latency and application errors before each step

   Lessons

   The failure and the later problems were not bugs in RDS Proxy; the application's connection pool implementation and settings were exposed by putting a proxy in between. With a proxy, backend connections are shared, so reuse depends on matching session state, closed transactions and connection lifetimes within the database's own timeouts. Before introducing it, the team should have checked what the application pool runs when it returns a connection, how it sets connection lifetime and what it runs on connect, more than the proxy settings themselves.

   View case
2. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage

   PlanetScale2025

   Symptoms

   At 7:13 UTC on 20 October 2025, PlanetScale's continuous tests in production started failing because they could not create new database branches. The service that creates, resizes and configures database branches was unavailable, a near-total control plane outage, while existing database branches kept their capacity and connectivity.

   Service outageScale-up failureNetwork unreachableHigh error rateReplication lagDNS resolution failures

   Trigger

   The incident started with a DNS misconfiguration at one of PlanetScale's service providers. An Amazon DynamoDB outage impacted AWS STS, and several hours of capacity constraints and network instability in AWS us-east-1 followed.

   Root cause

   Cloud provider outage

   Component

   DatabaseDatabase replicationDatabase proxy and connection poolerObject storage

   Contributing factor

   –

   Detection

   PlanetScale's continuous testing in production started failing and alerted engineers at 7:13 UTC. At 10:05 UTC an alert fired for a Kubernetes operator in a large single-tenant customer installation exhausting its resources.

   Remediation

   • Disallowed new databases in us-east-1 and made us-east-2 the default region for new databases
   • Delayed new backups and cancelled pending backups that were waiting for an EC2 instance
   • Advised PlanetScale Managed customers with vtgate autoscaling to shed load
   • Paused draining and terminating EC2 instances older than 30 days
   • Kept vacant EC2 instances for reuse instead of terminating them
   • Bin-packed vtgate processes more tightly for customers with autoscaling
   • Sent manual reparent requests to move primaries to healthier availability zones
   • Restarted the edge load balancer and vtgate processes that did not recover after the partitions healed

   Action items

   • Understand and become resilient to the failure modes of SaaS dependencies for CI/CD, SSO, web application hosting and incident communication
   • Investigate more ambitious ways to reduce runtime dependence on internal and AWS services
   • Look into using all six us-east-1 availability zones to tolerate zonal outages and partitions

   Lessons

   Keeping the control plane separate from the data plane meant the control plane outage did not affect customer databases, and redundancy, automated failover and zonal traffic routing limited the effect of the network partitions. Network partitions are hard to reason about, test and tolerate: with three availability zones a system survives losing one zone only if the other two can still reach each other reliably.

   View case
3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case
4. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022

   GitHub2022

   Symptoms

   On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

   Writes rejectedDatabase overloadHigh error rateFailover failure

   Trigger

   Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius

   Remediation

   • Failed over mysql1 to a healthy replica on March 16
   • Reset the application connections to the new primary on March 17
   • Added an index that fixed the main query performance problem
   • Performed a primary failover on March 22
   • Performed a primary failover on March 23
   • Throttled webhook traffic to reduce load

   Action items

   • Keep throttling webhook traffic at peak load times
   • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
   • Move traffic to other databases to reduce load and speed up failover
   • Review change management, particularly monitoring and changes during high load in production
   • Shard the databases and scale up hardware

   View case

By root cause4