Incident Library
Applied filters ComponentClient-side DNS resolver and cacheSymptomLoad imbalanceDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
    Symptoms

    In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

    Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance
    Trigger

    During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

    Root cause
    Capacity exhaustion
    Component
    conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver
    Contributing factor
    Unreviewed defaultsUnrepresentative stagingUnknown limit
    Remediation
    • Raised the conntrack maxPerCore value in the kube-proxy configuration
    • Installed dns-autoscaler so CoreDNS ran with more pods
    • Installed node-local-dns as a DaemonSet on every node
    • Replaced force_tcp with prefer_udp in the node-local-dns configuration
    Lessons

    On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

    View case
  2. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
    Symptoms

    On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

    High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures
    Trigger

    Several services sent a high volume of external DNS queries, all of which went through kube-dns.

    Root cause
    Capacity exhaustion
    Component
    kube-dnskube-proxyClient-side DNS resolver and cache
    Contributing factor
    Unreviewed defaults
    Detection

    Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

    Lessons

    By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

    View case
Check your cluster for this

These 2 cases involve Client-side DNS resolver and cache and kube-proxy. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Client-side DNS resolver and cache2kube-proxy2conntrack table1CoreDNS1kube-dns1VPC DNS resolver1
Contributing factor
Unreviewed defaults2Unknown limit1Unrepresentative staging1
Organisation
Airbnb1Chatwork1
  1. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
  2. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019