Incident Library

IL-0574 Safie · 2026-10-27 (published)awsdatabase

Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS Proxy

Safie's platform runs as microservices that read and write the same Aurora MySQL database, with a large share of write transactions. max_connections was already at its maximum of 16,000 on an instance class large enough for it, and connections kept rising with users, so the team put Amazon RDS Proxy in front of the database to multiplex many client connections onto fewer database connections. The settings were estimated from SHOW PROCESSLIST counts per client host. After about a week in staging, where the expected drop in connections did not show but the application worked, the change went to production.

Measured demand from the applications was about 7,600 connections, but MaxConnectionsPercent of 21% let the proxy open only about 3,360 connections to Aurora, so switching the endpoint cut the usable connections from 16,000 to 3,360 and about 4,200 connections had to wait to borrow one. Because ConnectionBorrowTimeout equalled the application's 7-second socket timeout, the applications timed out first, dropped the connection and opened a new one. Setting up new connections costs CPU in MySQL, and as CPU saturated, connection setup slowed further and more timeouts followed in a loop. The team rolled back at once by returning the ECS Fargate tasks that used RDS Proxy to their previous version.

The team reproduced the failure on a copy of Aurora restored from a staging snapshot, loaded with Locust, and confirmed that raising MaxConnectionsPercent to 62% removed it. RDS Proxy was then moved in step by step with feature flags in Flipt per ECS task, read requests first. Months later, with reads fully moved, the multiplexing worked but the Aurora writer's connections had risen by 5,000, from 6,400 by day and 4,200 at night to 11,700 and 8,900, because the proxy kept idle backend connections up to MaxIdleConnectionsPercent. Lowering that limit far enough would have meant reopening thousands of connections at the burst that comes at a fixed time every hour, close to the original failure, so Safie withdrew RDS Proxy.

Severity
–
Scope
The platform services that use the main Aurora MySQL database returned errors until the rollback
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

A few minutes after the production endpoint was switched to RDS Proxy, Safie's alert monitoring system fired on errors, and the CPU use of the Aurora MySQL writer was stuck near 100%.

In CloudWatch, new connection attempts (ConnectionAttempts), normally under 10,000, peaked at 264,000, and disconnected clients (AbortedClients) reached 258,000.

The first suspect was RDS Proxy connection pinning, but only about 1% of connections were pinned, which made it a weak explanation. Further investigation showed that the proxy's connection limit was less than half of the real connection demand.

Symptom Database overloadHigh error rate

Root cause

Root cause Configuration change

Component Database proxy and connection poolerDatabase
Contributing factor Retry stormUnrepresentative stagingNo staged rollout
Trigger
The production application endpoints were switched to the new RDS Proxy all at once, with MaxConnectionsPercent at 21%, MaxIdleConnectionsPercent at 10% and ConnectionBorrowTimeout at 7 seconds, the same as the application's socket timeout, after a week in staging had shown no problem.

Detection

Safie's alert monitoring system fired on errors a few minutes after the endpoint was switched to RDS Proxy.

Remediation

Action items

Lessons

The failure and the later problems were not bugs in RDS Proxy; the application's connection pool implementation and settings were exposed by putting a proxy in between. With a proxy, backend connections are shared, so reuse depends on matching session state, closed transactions and connection lifetimes within the database's own timeouts. Before introducing it, the team should have checked what the application pool runs when it returns a connection, how it sets connection lifetime and what it runs on connect, more than the proxy settings themselves.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database proxy and connection pooler, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.