IL-0445 Chatwork · 2021-01-05 (published)kubernetesawsnetworking
Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
Chatwork runs EKS as a single multi-tenant cluster. When traffic to the legacy system on EKS rose during the migration from EC2, conntrack tables on some nodes overflowed and dropped connections. Most entries came from CoreDNS, which was still at the EKS default of two pods. The conntrack limit turned out to be set by kube-proxy to a value other than the default, so Chatwork raised maxPerCore in the conntrack section of the kube-proxy configuration, which the post shows as max: 0, maxPerCore: 32768, min: 131072, tcpCloseWaitTimeout: 1h0m0s, tcpEstablishedTimeout: 24h0m0s.
At the same time Chatwork installed dns-autoscaler. The clusters it had built earlier with kube-aws included dns-autoscaler from the start, and the team had forgotten to add it to EKS. Once CoreDNS ran with enough pods, the cluster became much more stable. To stop DNS errors when CoreDNS pods were evicted from spot nodes, and to cut cluster-wide DNS requests and conntrack entries on the CoreDNS nodes, Chatwork added node-local-dns, which runs on every node as a DaemonSet.
With node-local-dns, production DNS became unstable again and node-local-dns logged frequent timeouts. The cause was force_tcp: AWS recommends not using force_tcp with CoreDNS on EC2, and a comment quoted from an amazon-vpc-cni-k8s issue says the software forwarding DNS requests on EC2 Nitro instances is limited to 2 simultaneous TCP connections and blocks on TCP queries. Chatwork replaced force_tcp with prefer_udp, and DNS became stable. The final setup was dns-autoscaler plus node-local-dns without force_tcp. Because node-local-dns redirects requests for kube-dns to itself with a dummy device, application DNS requests go straight to CoreDNS when node-local-dns is temporarily unavailable, for example during its update.
- Severity
- –
- Scope
- The legacy system being moved to EKS: connections to the application were dropped and applications got DNS errors
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.
Most entries in the conntrack tables were CoreDNS traffic. The cluster still ran the EKS default of two CoreDNS pods, so DNS requests piled onto them and filled the conntrack tables of their nodes, and the CPU of both CoreDNS pods was also close to full. While DNS was unstable, DNS queries blocked, and the number of pods grew more than needed.
After the CoreDNS pods were scaled out, the application still got DNS errors whenever a CoreDNS pod went away, which happened regularly because Chatwork's node groups run entirely on spot instances. Judging from the timing, in-flight DNS requests seemed to fail when a CoreDNS pod was evicted.
When node-local-dns was then applied from the documented manifest, after checking it in test and staging, DNS became unstable again in production, which has far more requests, and the node-local-dns logs showed frequent timeouts. The team had believed node-local-dns would not be used until node groups were created with eksctl settings pointing pods' DNS at it, so the timeouts were unexpected; in fact node-local-dns creates a dummy device and iptables rules at startup and takes over requests sent to kube-dns.
Symptom Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance
Root cause
Root cause Capacity exhaustion
- Trigger
- During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.
Detection
–
Remediation
- Raised the conntrack maxPerCore value in the kube-proxy configuration
- Installed dns-autoscaler so CoreDNS ran with more pods
- Installed node-local-dns as a DaemonSet on every node
- Replaced force_tcp with prefer_udp in the node-local-dns configuration
Action items
–
Lessons
On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.
Sources
References
Related cases
IL-0048DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025same root cause · shared CoreDNS, VPC DNS resolver, Client-side DNS resolver and cache
IL-0027All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared CoreDNS, Client-side DNS resolver and cache
IL-0433Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errorsMonotaRO · 2023same root cause · shared CoreDNS, Client-side DNS resolver and cache
IL-0013The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared kube-proxy, conntrack table
IL-0033Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019same root cause · shared kube-proxy, Client-side DNS resolver and cache
IL-0001Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared CoreDNS, Client-side DNS resolver and cache
IL-0102CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared CoreDNS
IL-0155Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared CoreDNS
IL-0082Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared CoreDNS
IL-0170During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025same root cause · shared conntrack table
IL-0204An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API serverRender · 2024same root cause · shared CoreDNS
IL-0440Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packetsCybozu · 2024same root cause · shared Client-side DNS resolver and cache
IL-0182Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation이언철 · 2024same root cause · shared CoreDNS
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0445
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0445/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.