Incident Library
Applied filters Root causeDNS or network faultFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019
    Symptoms

    Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.

    Intermittent timeoutsHigh latencyDNS resolution failures
    Trigger

    The C library resolver sent the A and AAAA queries for each lookup on the same port; where DNAT or SNAT translation takes place, a race condition in conntrack sometimes left one of the queries unanswered, and the resolver waited 5 seconds before retrying.

    Root cause
    DNS or network fault
    Component
    conntrack tableClient-side DNS resolver and cacheLinux kernel
    Contributing factor
    Unreviewed defaults
    Detection

    The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.

    Remediation
    • added the single-request-reopen option to dnsConfig in the deployment specs
    • set GRPC_DNS_RESOLVER=native in the Dockerfiles of gRPC applications so they used the libc resolver
    • checked with tcpdump port 53 that A and AAAA queries left from different ports
    Lessons

    Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.

    View case
Check your cluster for this

These 6 cases involve conntrack table and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

DNS or network fault6 cases

Component
conntrack table3Client-side DNS resolver and cache2Linux kernel2AWS VPC CNI2Cloud load balancer1CoreDNS1Horizontal pod autoscaler1Ingress controller1kube-proxy1NCCL1NIC, cable and switch port link1Node operating system image1Pod termination1Telemetry agent1
Contributing factor
Unreviewed defaults6Unverified assumption2Ignored early signal1Unpatched known bug1Unknown limit1
Organisation
MindTickle2Adevinta1ByteDance1Lotte E-commerce1Prezi1
  1. IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUs
    ByteDance2024
  2. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  3. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  4. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020
  5. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019
  6. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019