Cases1 case

1 to 1 of 1

1. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped

   Chatwork2021

   Symptoms

   In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

   Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance

   Trigger

   During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unreviewed defaultsUnrepresentative stagingUnknown limit

   Remediation

   • Raised the conntrack maxPerCore value in the kube-proxy configuration
   • Installed dns-autoscaler so CoreDNS ran with more pods
   • Installed node-local-dns as a DaemonSet on every node
   • Replaced force_tcp with prefer_udp in the node-local-dns configuration

   Lessons

   On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

   View case

By root cause1