IL-0150 Amazon Web Services · 2025-10-19awskubernetesnetworking
DynamoDB keeps hundreds of thousands of DNS records for its load balancers in each region, maintained by automation in two parts. The DNS Planner watches load balancer health and capacity and produces new DNS plans; the DNS Enactor, running independently in three availability zones, applies each plan to Route 53 in a transaction. Before applying a plan, an Enactor checks once that it is newer than the plan already applied. That night one Enactor was badly delayed by retries while the Planner kept producing new plans and a second Enactor applied a much newer one. When the second Enactor finished, its clean-up deleted plans far older than its own. At the same moment the delayed Enactor, whose newer-than check was by now stale, wrote its old plan over the regional endpoint; the clean-up then deleted that plan, leaving the record for dynamodb.us-east-1.amazonaws.com empty. With the active plan gone, no Enactor could apply later plans, and the automation could not repair itself.
Engineers identified DynamoDB's DNS state as the source at 12:38 AM. By 1:15 AM temporary mitigations let some internal services connect and repaired internal tooling, and manual operator intervention restored all DNS information by 2:25 AM. Clients recovered as cached DNS records expired between 2:25 and 2:40 AM.
EC2 took much longer. The DropletWorkflow Manager (DWFM), which holds a lease on every physical server (droplet) that hosts instances, depends on DynamoDB for its periodic state checks, so leases across the fleet had slowly been timing out. When DynamoDB returned, DWFM tried to re-establish leases with so many droplets that the work timed out and was queued again, a congestive collapse with no established recovery procedure. At 4:14 AM engineers throttled incoming work and selectively restarted DWFM hosts, and by 5:28 AM all leases were back. The Network Manager then had a large backlog of network configuration for new instances, so new instances lacked connectivity until 10:36 AM. NLB's health checks brought those instances into service before their network state had propagated, which made health checks flap and triggered automatic availability zone DNS failover that removed capacity; engineers disabled automatic health check failover at 9:36 AM. Throttles were relaxed from 11:23 AM and EC2 was fully recovered at 1:50 PM.
ECS, EKS and Fargate in us-east-1 had container launch failures and cluster scaling delays from 11:45 PM PDT on 19 October until the services recovered at 2:20 PM PDT on 20 October.
At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.
EC2 API calls in us-east-1 returned more errors and took longer, and new instance launches failed with "insufficient capacity" errors, later also with "request limit exceeded" errors from throttling. Instances already running were not affected. From 5:28 AM new launches began to succeed, but new instances could lack network connectivity until about 10:36 AM.
From 5:30 AM some Network Load Balancers had more connection errors: health checks alternated between failing and passing, so NLB nodes and targets were removed from DNS and put back. Lambda, Amazon Connect, STS, IAM console sign-in, Redshift and AWS Support were also affected.
Between 11:45 PM PDT on 19 October and 2:20 PM PDT on 20 October, ECS, EKS and Fargate in us-east-1 had container launch failures and cluster scaling delays.
Symptom DNS resolution failuresHigh error rateScale-up failure
Root cause Software defect
DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.
A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result.
Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.
IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the loadGoogle Cloud · 2024same root cause · shared Cloud load balancer
IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Application database
IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404DFDS · 2021same root cause · shared Cloud load balancer
IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Application database
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Cloud load balancer