Incident Library
Applied filters Componentconntrack tableSymptomLoad imbalanceFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021
    Symptoms

    In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

    Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance
    Trigger

    During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

    Root cause
    Capacity exhaustion
    Component
    conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver
    Contributing factor
    Unreviewed defaultsUnrepresentative stagingUnknown limit
    Remediation
    • Raised the conntrack maxPerCore value in the kube-proxy configuration
    • Installed dns-autoscaler so CoreDNS ran with more pods
    • Installed node-local-dns as a DaemonSet on every node
    • Replaced force_tcp with prefer_udp in the node-local-dns configuration
    Lessons

    On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

    View case
Check your cluster for this

This case involves conntrack table and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
conntrack tableCoreDNSClient-side DNS resolver and cachekube-proxyVPC DNS resolver
Contributing factor
Unreviewed defaultsUnknown limitUnrepresentative staging
Organisation
Chatwork
  1. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped
    Chatwork2021