Incident Library
Applied filters Componentconntrack tableClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019
    Symptoms

    Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.

    Intermittent timeoutsHigh latencyDNS resolution failures
    Trigger

    The C library resolver sent the A and AAAA queries for each lookup on the same port; where DNAT or SNAT translation takes place, a race condition in conntrack sometimes left one of the queries unanswered, and the resolver waited 5 seconds before retrying.

    Root cause
    DNS or network fault
    Component
    conntrack tableClient-side DNS resolver and cacheLinux kernel
    Contributing factor
    Unreviewed defaults
    Detection

    The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.

    Remediation
    • added the single-request-reopen option to dnsConfig in the deployment specs
    • set GRPC_DNS_RESOLVER=native in the Dockerfiles of gRPC applications so they used the libc resolver
    • checked with tcpdump port 53 that A and AAAA queries left from different ports
    Lessons

    Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.

    View case
Check your cluster for this

These 8 cases involve conntrack table and kube-proxy. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Software defect1 case

Component
conntrack tableCoreDNSIstiokube-proxy
Contributing factor
Unreviewed defaults
Organisation
Cisco ThousandEyes
  1. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout
    Cisco ThousandEyes2025

Capacity exhaustion2 cases

Component
conntrack table2HAProxy1Ingress controller1Istio1kube-proxy1Node pool1
Contributing factor
Unreviewed defaults1Unknown limit1Monitoring blind spot1No alert on limit1No capacity planning1Single point of failure1
Organisation
Backpackr1loveholidays1
  1. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  2. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020

DNS or network fault4 cases

Component
conntrack table4kube-proxy2Linux kernel2Cloud load balancer1CoreDNS1Client-side DNS resolver and cache1Horizontal pod autoscaler1Ingress controller1Node operating system image1Pod termination1AWS VPC CNI1
Contributing factor
Unreviewed defaults3Automation amplification1Ignored early signal1Unpatched known bug1No caching1
Organisation
Lotte E-commerce1MindTickle1Preply1Prezi1
  1. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
  3. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019
  4. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019

Configuration change1 case

Component
Cloud load balancerconntrack tableHAProxyIngress controller
Contributing factor
Unverified assumptionExcessive blast radiusChange with a delayed effectRetry storm
Organisation
LINE+
  1. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE+2022