Cases2 cases

1 to 2 of 2

1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsApplication databaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx

   LINE+2022

   Symptoms

   The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.

   Connection resetsDatabase overloadIntermittent connection failuresHigh latency

   Trigger

   Another service that shared the same L7 load balancer deployed its own load balancer configuration in the Verda dashboard, which reloaded HAProxy; 30 minutes later, at the hard-stop-after limit, the old HAProxy process exited and closed every WebSocket connection it still held.

   Root cause

   Configuration change

   Component

   HAProxyCloud load balancerconntrack tableIngress controller

   Contributing factor

   Change with a delayed effectExcessive blast radiusRetry stormUnverified assumption

   Detection

   The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.

   Remediation

   • separated the Merchant system's token issuance API into its own deployment so reconnect traffic did not delay other Merchant APIs
   • scaled out Nginx with pod anti-affinity and a label selector so each pod ran on a different node, which stopped the nf_conntrack packet drops
   • raised Ingress Nginx proxy-read-timeout and proxy-send-timeout and the load balancer timeout to 3600s, which did not stop the drops
   • moved messaging-hub from the shared L7 load balancer to an L4 load balancer, which ended the WebSocket drops

   Lessons

   A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days. A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.

   View case

By root cause2