IL-0433 MonotaRO · 2023-08-24 (published)kubernetesawsnetworking
Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
MonotaRO was moving a monolithic backend API with more than 200 endpoints from EC2 to EKS, with Fargate for the nodes and Datadog and Sentry for monitoring. The ALB shifted the share of requests per endpoint step by step. After some endpoints had been moved, raising the rest from 1% to 25% produced mass name resolution errors, and the traffic was moved back to the EC2 version. The two CoreDNS pods had a 100m CPU request and a 170Mi memory limit, so Fargate ran each on a 0.25 vCPU machine, and the load on them exceeded the request. An HPA for CoreDNS was added as a stopgap and the migration was completed with it, but it raised running cost and scale-in still caused occasional resolution errors.
The CoreDNS logs showed each lookup of example.com first trying example.com.default.svc.cluster.local, example.com.svc.cluster.local and example.com.cluster.local for both A and AAAA records, each answered NXDOMAIN, before example.com. answered NOERROR. The pod's /etc/resolv.conf had options ndots:5, so any name with fewer than five dots was tried with every search domain first. The CoreDNS cache TTL had never been set and was the default of 5 seconds, shorter than the TTL of the DNS servers behind CoreDNS. The occasional errors on scale-in came from CoreDNS pods being cut off during termination.
The permanent fixes were dnsConfig with ndots set to "1" in the application Deployments, cache 600 in the Corefile in the coredns ConfigMap (CoreDNS reloaded the changed Corefile by itself without a pod restart), and lameduck 30s in the health plugin so a terminating CoreDNS pod keeps serving for 30 seconds. A preStop sleep was not possible because the CoreDNS image is a distroless image from Google with no shell.
- Severity
- –
- Scope
- Requests to the backend API on EKS failed to reach the database and outside services through the proxy, so the traffic shift was rolled back
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.
The API is written in Python. Connections through the proxy failed with NewConnectionError('<urllib3.connection.HTTPSConnection object at 12345678>: Failed to establish a new connection: [Errno -3] Temporary failure in name resolution'), and database connections failed with OperationalError:(2005, "Unknown MySQL server host 'db.example.com' (-3)"). The switch was done at a quiet time, at about 1000 RPS against a peak of 2000 to 3000 RPS.
Name resolution in the cluster is done by CoreDNS, so the team suspected CoreDNS first. After moving the traffic back, the CoreDNS CPU and memory use showed load above the 100m CPU request in the CoreDNS pod spec. With that small request, Fargate had provisioned each CoreDNS pod on a 0.25 vCPU machine, and the Deployment had replicas: 2, so only two CoreDNS pods ran.
As a stopgap the team autoscaled CoreDNS with an HPA, which let the migration finish, but name resolution errors still appeared from time to time when the HPA scaled CoreDNS in.
Root cause
Root cause Capacity exhaustion
- Trigger
- MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.
Detection
Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.
Remediation
- Shifted the requests back to the EC2 version to stop the errors
- Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish
Action items
- Set ndots to 1 through dnsConfig in the application Deployments
- Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
- Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
- Consider reducing the differences between production and staging, weighed against cost
Lessons
MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.
Sources
Related cases
IL-0102CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared CoreDNS, Horizontal pod autoscaler
IL-0181A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed RunningKevin · 2026same root cause · shared Horizontal pod autoscaler, Container CPU limit
IL-0048DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025same root cause · shared CoreDNS, Client-side DNS resolver and cache
IL-0027All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared CoreDNS, Client-side DNS resolver and cache
IL-0182Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared CoreDNS, Horizontal pod autoscaler
IL-0445Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were droppedChatwork · 2021same root cause · shared CoreDNS, Client-side DNS resolver and cache
IL-0001Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared CoreDNS, Client-side DNS resolver and cache
IL-0155Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared CoreDNS
IL-0082Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared CoreDNS
IL-0204An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0440Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packetsCybozu · 2024same root cause · shared Client-side DNS resolver and cache
IL-0028Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023same root cause · shared Container CPU limit
IL-0033Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019same root cause · shared Client-side DNS resolver and cache
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0433
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0433/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.