Cases3 cases

1 to 3 of 3

1. IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS Proxy

   Safie2026

   Symptoms

   A few minutes after the production endpoint was switched to RDS Proxy, Safie's alert monitoring system fired on errors, and the CPU use of the Aurora MySQL writer was stuck near 100%.

   Database overloadHigh error rate

   Trigger

   The production application endpoints were switched to the new RDS Proxy all at once, with MaxConnectionsPercent at 21%, MaxIdleConnectionsPercent at 10% and ConnectionBorrowTimeout at 7 seconds, the same as the application's socket timeout, after a week in staging had shown no problem.

   Root cause

   Configuration change

   Component

   Database proxy and connection poolerDatabase

   Contributing factor

   Retry stormUnrepresentative stagingNo staged rollout

   Detection

   Safie's alert monitoring system fired on errors a few minutes after the endpoint was switched to RDS Proxy.

   Remediation

   • Rolled back the ECS Fargate tasks that used RDS Proxy to their previous version

   Action items

   • Raise MaxConnectionsPercent to 62% and MaxIdleConnectionsPercent to 31%
   • Choose between direct Aurora and RDS Proxy with a feature flag evaluated per ECS task, so the switch can be made in stages and undone without a deployment
   • Move read requests first and write requests afterwards
   • Raise the share in steps of 1%, 10%, 25%, 50%, 75% and 100%, checking proxy borrow wait, new backend connections, Aurora connections, CPU and latency and application errors before each step

   Lessons

   The failure and the later problems were not bugs in RDS Proxy; the application's connection pool implementation and settings were exposed by putting a proxy in between. With a proxy, backend connections are shared, so reuse depends on matching session state, closed transactions and connection lifetimes within the database's own timeouts. Before introducing it, the team should have checked what the application pool runs when it returns a connection, how it sets connection lifetime and what it runs on connect, more than the proxy settings themselves.

   View case
2. IL-0561 DigitalOcean's Cloud Control Panel and Public API returned up to 25% errors in two windows as authorization timed out

   DigitalOcean2026

   Symptoms

   From 16:56 UTC on 24 August 2026, requests to DigitalOcean's Cloud Control Panel, Public API and the product management systems that depend on them failed at elevated rates. Automated monitoring flagged abnormal error rates and latency at 16:58. Running Droplets, network routing and block storage Volumes kept working.

   High error rateHigh latency

   Trigger

   A planned migration of the database layer behind DigitalOcean's identity, authentication and authorization services made one critical authorization query run much slower than its historical baseline.

   Root cause

   Configuration change

   Component

   DatabaseIdentity and access service

   Contributing factor

   Retry stormHarmful manual action

   Detection

   Automated monitoring detected abnormal error rates and latency at 16:58 UTC, two minutes after the authorization query slowed down.

   Remediation

   • Rerouted authorization traffic onto a single healthy processing path
   • Temporarily disabled hedged retries for the primary callers
   • Disabled the retry and amplification behaviour across the remaining affected services
   • Added read capacity for the slow authorization query

   Action items

   • Deploy circuit breakers in upstream services that detect a degraded downstream dependency and suppress hedged requests and retries
   • Audit the architecture so retry suppression and circuit breaking are enforced the same way in every control plane service
   • Add fast-fail load shedding so request queues do not saturate while a dependency is slow
   • Add telemetry alerts that spot queues backing up before customers are affected
   • Let critical authorization components run in a degraded mode on asynchronous, eventually consistent database replicas when the primary database is slow

   View case
3. IL-0283 Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature file

   Cloudflare2025

   Symptoms

   From about 11:20 UTC on 18 November 2025, Cloudflare's network started failing to deliver core traffic, and Internet users visiting Cloudflare customers' sites got an error page that pointed to a failure inside Cloudflare's network. The volume of HTTP 5xx responses served by the network jumped far above its normal low baseline.

   High error rateService outageHigh latencyResponse quality degradation

   Trigger

   At 11:05 UTC Cloudflare deployed a ClickHouse access control change that made users' existing implicit access to the underlying tables in the r0 database explicit, so the metadata of those tables became visible in system tables such as system.columns. The change was being rolled out gradually across the cluster.

   Root cause

   Configuration change

   Component

   DatabaseCDN and edge network

   Contributing factor

   Unverified assumptionPartial rolloutNo staged rolloutNo configuration testsOverlapping failuresRetry storm

   Detection

   The first automated test detected the issue at 11:31 UTC, three minutes after the first errors on customer HTTP traffic at 11:28. Manual investigation started at 11:32 and the incident call was created at 11:35.

   Remediation

   • Bypassed the core proxy for Workers KV and Access so they fell back to a prior proxy version, which reduced their impact
   • Stopped the automatic creation and deployment of new Bot Management feature files
   • Inserted a known good feature file into the distribution queue and deployed it globally
   • Forced a restart of the core proxy
   • Scaled control plane concurrency to clear the backlog of dashboard logins
   • Restarted the remaining services that had entered a bad state

   Action items

   • Harden the ingestion of Cloudflare-generated configuration files the same way as user-generated input
   • Enable more global kill switches for features
   • Stop core dumps and other error reports from being able to overwhelm system resources
   • Review the failure modes for error conditions across all core proxy modules

   View case

By root cause1