Cases17 cases

2 to 6 of 17

1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsDatabaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down

   Backpackr2025

   Symptoms

   Services on the production Amazon EKS cluster behind the idus marketplace threw intermittent errors whose cause was hard to pin down. Calls to external services such as RDS and Redis failed now and then, and whenever DNS traffic spiked, domain lookups answered slowly or failed outright.

   DNS resolution failuresIntermittent connection failuresHigh latencyOOMKilledService outage

   Trigger

   All CoreDNS pods were scheduled onto the same node, and that node's memory usage spiked until it ran out of memory.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cachekubeletContainer memory limitNode pool

   Contributing factor

   Unreviewed defaultsSingle point of failureNo caching

   Detection

   The write-up does not say how the full outage was noticed. The earlier DNS trouble showed up as intermittent service errors, and investigating failed calls to RDS and Redis led to DNS resolution failures.

   Remediation

   • deployed NodeLocal DNSCache as a DaemonSet in front of CoreDNS
   • added a PodDisruptionBudget (maxUnavailable 1) and topologySpreadConstraints across zones for CoreDNS
   • lowered ndots from 5 to 2 with dnsConfig in the Deployments

   Lessons

   The CoreDNS setup that EKS creates by default (two pods) can be enough for a small or development cluster but becomes a bottleneck in a large production cluster, and DNS is a single point of failure that needs high availability and room to scale. Tuning the CoreDNS cache and buffer size only relieved the symptoms; the real load came from the extra queries that ndots:5 generates for external names. A CoreDNS pod without a memory limit can share a node's fate with everything else on it.

   View case
3. IL-0084 Packet drops on a bare metal provider's vSwitch forced Namespace to rebuild its fra3 region and left global capacity short, and the next day slow NVMe writes on a control plane node drove the Kubernetes API server to very high CPU

   Namespace2024

   Symptoms

   From 17 October 2024, packets through the vSwitch that joined Namespace's bare metal machines at one provider in the fra3 region were dropped at random. Retries in TCP, QUIC and the Namespace control plane hid most of it, but the SLO error budget for latency targets on key interactions began to burn.

   Intermittent connection failuresHigh latencyControl plane lagService outage

   Trigger

   Random packet drops on the bare metal provider's vSwitch from 17 October 2024; for the second outage, a rate of Kubernetes changes that the slow NVMe on one control plane node could no longer keep up with.

   Root cause

   Cloud provider outage

   Component

   Kubernetes API server

   Contributing factor

   No capacity planningNo headroomOverlapping failures

   Detection

   SLO-based monitoring of latency targets showed the error budget burning while retries hid the individual packet drops.

   Remediation

   • diverted all customer traffic from fra3 to other regions
   • connected each provider machine to the zrh1 cluster over Wireguard instead of the vSwitch
   • repaved every fra3 machine with a new identity, certificates and encryption-at-rest keys, dropping regional caches to recover faster
   • cloned existing datasets in a more distributed way to speed up repaving the machines
   • added machines back to the new region in batches and returned the region to the compute pool
   • redirected workloads away from zrh1 during the second outage
   • cloned the failing control plane node's key state to a higher-capacity node that took over its role and IP address

   Action items

   • Credit all instances that ran during the outages back to customers
   • Build up backup regions and solutions so a similar event has options that do not affect customers

   Lessons

   Running hot with a delayed hardware delivery left no room to take a region out, so fixing a provider network fault turned into a global capacity shortage. After one outage the team was anchored on its cause and first looked for network problems in the second outage, which had a different cause: a control plane node whose disk could not keep up with the Kubernetes change rate.

   View case
4. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled

   DigitalOcean2024

   Symptoms

   From 22:25 to 22:50 UTC every DigitalOcean service in the SFO3 region had network connectivity problems, as traffic between the datacenter and the backbone dropped. Traffic through the router was back by 22:30 and SFO3's control plane systems had reconnected by 22:50, and most services recovered then.

   Node NotReadyControl plane unresponsiveIntermittent connection failures

   Trigger

   Most of the line cards on one SFO3 core router rebooted at the same time at 22:21 UTC because of hardware errors on the device.

   Root cause

   DNS or network fault

   Component

   Kubernetes API serverkubelet

   Contributing factor

   –

   Remediation

   • The crashed line cards came back online and traffic on the core router was restored
   • Manually recycled all affected nodes across the SFO3 DOKS fleet

   Action items

   • Work with the router vendor's support team to find the root cause of the line card crash
   • Upgrade software on the core routers in SFO3
   • Raise thresholds for automated remediation so affected nodes are restored without manual action

   Lessons

   A short network fault can leave Kubernetes nodes broken long after the network itself is healthy, when the guest networking on the machines does not recover by itself. Checking node health and the API servers after the network returns, and having automated remediation that is allowed to replace many nodes at once, shortens that tail.

   View case
5. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation

   이언철2024

   Symptoms

   When traffic changed sharply, CoreDNS came under heavy load, and the cluster saw network problems and a sudden rise in latency. The horizontal pod autoscaler then added pods, database connections surged, and the impact spread to the whole service.

   High latencyIntermittent connection failuresAutoscaler misbehaviour

   Trigger

   Sharp changes in traffic loaded the small number of CoreDNS pods beyond what they could handle.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSKarpenterHorizontal pod autoscalerPod scheduling and placement

   Contributing factor

   Automation amplificationHarmful manual action

   Remediation

   • Set up the Cluster Proportional Autoscaler for CoreDNS with nodesPerReplica and coresPerReplica chosen for the instance types in use
   • Spread CoreDNS with soft topologySpreadConstraints (ScheduleAnyway) instead of anti-affinity so Karpenter could consolidate nodes
   • Added alert thresholds on CoreDNS error rate, panics, request duration and request rate
   • Deployed NodeLocal DNSCache so each node answers DNS from a local cache and forwards misses to the main CoreDNS

   Lessons

   A fixed, small number of CoreDNS pods cannot follow a cluster whose nodes and pods change quickly with traffic, and the damage spreads: DNS latency triggers autoscaling, which in turn floods the database with connections. Simply adding many CoreDNS replicas created a different problem, because with tainted nodes and a maxSkew: 1 spread constraint the pods piled onto a few nodes and blocked Karpenter consolidation. Scaling CoreDNS in proportion to the cluster and putting a DNS cache on every node handled both.

   View case

By root cause6