Cases3 cases

1 to 3 of 3

1. IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS Proxy

   Safie2026

   Symptoms

   A few minutes after the production endpoint was switched to RDS Proxy, Safie's alert monitoring system fired on errors, and the CPU use of the Aurora MySQL writer was stuck near 100%.

   Database overloadHigh error rate

   Trigger

   The production application endpoints were switched to the new RDS Proxy all at once, with MaxConnectionsPercent at 21%, MaxIdleConnectionsPercent at 10% and ConnectionBorrowTimeout at 7 seconds, the same as the application's socket timeout, after a week in staging had shown no problem.

   Root cause

   Configuration change

   Component

   Database proxy and connection poolerDatabase

   Contributing factor

   Retry stormUnrepresentative stagingNo staged rollout

   Detection

   Safie's alert monitoring system fired on errors a few minutes after the endpoint was switched to RDS Proxy.

   Remediation

   • Rolled back the ECS Fargate tasks that used RDS Proxy to their previous version

   Action items

   • Raise MaxConnectionsPercent to 62% and MaxIdleConnectionsPercent to 31%
   • Choose between direct Aurora and RDS Proxy with a feature flag evaluated per ECS task, so the switch can be made in stages and undone without a deployment
   • Move read requests first and write requests afterwards
   • Raise the share in steps of 1%, 10%, 25%, 50%, 75% and 100%, checking proxy borrow wait, new backend connections, Aurora connections, CPU and latency and application errors before each step

   Lessons

   The failure and the later problems were not bugs in RDS Proxy; the application's connection pool implementation and settings were exposed by putting a proxy in between. With a proxy, backend connections are shared, so reuse depends on matching session state, closed transactions and connection lifetimes within the database's own timeouts. Before introducing it, the team should have checked what the application pool runs when it returns a connection, how it sets connection lifetime and what it runs on connect, more than the proxy settings themselves.

   View case
2. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once

   Discord2026

   Symptoms

   From 12:13 to 15:30 PDT on 25 March 2026, voice and video on Discord were heavily degraded. Users mostly could not start or join calls and saw "Awaiting Endpoint" in their call status.

   Service outage

   Trigger

   A routine change to tune the sessions service cluster on Kubernetes gave each pod more CPU and memory and lowered the pod count in proportion. When it was deployed to the first zone at 12:13 PDT, Kubernetes terminated 50% of that zone's pods, and the termination grace period ran out before the sessions could be handed off.

   Root cause

   Configuration change

   Component

   Pod terminationCoordination and key-value storeDatabase proxy and connection pooler

   Contributing factor

   Retry stormUnknown limit

   Remediation

   • Set low syncer spawn rate limits in the calls, streams and guilds services and restarted the voice syncers cluster
   • Restarted voice syncers instances one at a time, and four of the first five recovered fully
   • Brought 15 new voice syncers instances online, doubling capacity and halving the syncers per instance
   • Restarted the last backed-up voice syncers instance, leaving the doubled cluster fully healthy

   Action items

   • Add a validating admission webhook that rejects a scale-down of Elixir workloads until each pod has handed off its processes
   • Replace the Holster.Pool supervisor with a PartitionSupervisor so outgoing connections are created and checked out concurrently
   • Move gun connection process lifecycle management into Holster.Pool so the gun connection supervisor is no longer used
   • Tune the upstream rate limits that call into voice syncers, with load shedding tooling that can force them
   • Add rate limits on endpoint selection in voice syncers to limit outbound connection creation
   • Monitor the HTTP connection pooling mailboxes
   • Review and expand instrumentation and logging for service discovery announcement and voice syncer RPCs to SFUs
   • Monitor the most critical processes across all services instead of aggregate metrics such as overall mailbox health
   • Bring more workloads into Kubernetes so capacity can be scaled with a single command instead of provisioning VMs by hand
   • Re-architect the A/V system to scale its control plane services horizontally across multiple regions

   Lessons

   Discord says a large enough traffic spike will find a bottleneck somewhere; in earlier large-load incidents its ScyllaDB databases limited recovery, and this time the bottleneck moved to the voice services. Facing a capacity constraint, the team asks whether to increase supply, such as adding services, or reduce demand, such as rate limiting inbound traffic. During an incident it matters to know the toolbox, and Elixir and the BEAM give engineers many introspection options. Any slowdown in the hot loop of a single-threaded process can degrade both the service and the user experience.

   View case
3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case

By root cause2