IL-0561 DigitalOcean · 2026-08-24database
DigitalOcean's Cloud Control Panel and Public API returned up to 25% errors in two windows as authorization timed out
DigitalOcean was migrating the database layer that supports its identity, authentication and authorization services as planned infrastructure work. At 16:56 UTC the migration made a critical authorization query much slower. Several upstream services reacted to the slow query with automated hedged retries, which amplified the requests reaching the authentication service. The extra requests built a queue larger than the authentication service pool could process, and requests along the authorization path timed out in a cascade. Requests to the Cloud Control Panel, the Public API and other management control planes failed, while the data plane stayed up because established connections do not depend on the authorization path.
To end the first window, engineers sent authorization traffic over a single healthy processing path and switched off hedging for the main callers, and service came back. At 21:30 traffic was returned to the standard multipath routing. The query was still slow, and callers that did not hedge still retried, so the queue saturated again and the second window began. Teams then disabled retry and amplification behaviour in the remaining affected services and added read capacity for the slow query, and the service was stable from 23:09. The database migration was completed afterwards, and no customer data was lost.
- Severity
- –
- Scope
- Cloud Control Panel, Public API, Kubernetes cluster authentication, Container Registry and Serverless / AI Inference endpoints failed requests at about 10% on average and 25% at peak; Droplets, network routing and Volumes were not affected
- Users affected
- fewer than 2% of active customer accounts saw at least one failed API or Control Panel request
- Region
- –
- Time to detect
- 2m
- Time to resolve
- 6h 13m
Symptoms
From 16:56 UTC on 24 August 2026, requests to DigitalOcean's Cloud Control Panel, Public API and the product management systems that depend on them failed at elevated rates. Automated monitoring flagged abnormal error rates and latency at 16:58. Running Droplets, network routing and block storage Volumes kept working.
The status page told users to expect elevated 5xx errors for all API requests, including creates for various products, and said cloud.digitalocean.com might not load. Serverless Inference users saw 403 and 503 errors on their API requests. The affected management control planes were the Cloud Control Panel, the Public API, Kubernetes cluster authentication, Container Registry and the Serverless / AI Inference endpoints.
The errors came in two windows that added up to 3.5 hours of degradation within about 6 hours. The first window ran from 16:56 until about 18:52, when operators had tuned retries and moved requests onto one healthy service instance. At 21:30, moving traffic back to the normal production pattern brought the same saturation back, and at 22:27 the status page reported a recurrence. The second window lasted until 23:09 and reached the peak error rate of 25%. Across both windows about 10% of authenticated requests failed on average.
Symptom High error rateHigh latency
Root cause
Root cause Configuration change
- Trigger
- A planned migration of the database layer behind DigitalOcean's identity, authentication and authorization services made one critical authorization query run much slower than its historical baseline.
Detection
Automated monitoring detected abnormal error rates and latency at 16:58 UTC, two minutes after the authorization query slowed down.
Remediation
- Rerouted authorization traffic onto a single healthy processing path
- Temporarily disabled hedged retries for the primary callers
- Disabled the retry and amplification behaviour across the remaining affected services
- Added read capacity for the slow authorization query
Action items
- Deploy circuit breakers in upstream services that detect a degraded downstream dependency and suppress hedged requests and retries
- Audit the architecture so retry suppression and circuit breaking are enforced the same way in every control plane service
- Add fast-fail load shedding so request queues do not saturate while a dependency is slow
- Add telemetry alerts that spot queues backing up before customers are affected
- Let critical authorization components run in a degraded mode on asynchronous, eventually consistent database replicas when the primary database is slow
Lessons
–
Sources
- https://status.digitalocean.com/incidents/4qwm6pzlx1z3 · official-status · archived copy
Related cases
IL-0574Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS ProxySafie · 2026same root cause · shared Database
IL-0366Cloud Storage uploads and writes in us-central1 slowed and failed with HTTP 503 as traffic piled onto a few backendsGoogle Cloud · 2026same root cause · shared Database
IL-0179With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second burstsMUSINSA · 2026same root cause · shared Database
IL-0487Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexedAutumn · 2026same root cause · shared Database
IL-0283Cloudflare's core proxy returned HTTP 5xx across its network after loading an oversized Bot Management feature fileCloudflare · 2025same root cause · shared Database
IL-0528Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hoursGoogle Cloud · 2025same root cause · shared Database
IL-0392MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024same root cause · shared Database
IL-0367Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on themMicrosoft Azure · 2024same root cause · shared Database
IL-0511MIT xPro stopped responding after a configuration deploy as pgbouncer logins to RDS failed with 'wrong password type'MIT Open Learning · 2023same root cause · shared Database
IL-0331An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020same root cause · shared Database
IL-0272New Cassandra nodes served reads for data they did not hold, so Monzo records looked missing and payments failedMonzo · 2019same root cause · shared Database
IL-0292App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be readGoogle Cloud · 2017same root cause · shared Database
IL-0325DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deletedDigitalOcean · 2017same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0561
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0561/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.