Incident Library
Applied filters Root causeDNS or network faultClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019
    Symptoms

    Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.

    Intermittent timeoutsHigh latencyDNS resolution failures
    Trigger

    The C library resolver sent the A and AAAA queries for each lookup on the same port; where DNAT or SNAT translation takes place, a race condition in conntrack sometimes left one of the queries unanswered, and the resolver waited 5 seconds before retrying.

    Root cause
    DNS or network fault
    Component
    conntrack tableClient-side DNS resolver and cacheLinux kernel
    Contributing factor
    Unreviewed defaults
    Detection

    The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.

    Remediation
    • added the single-request-reopen option to dnsConfig in the deployment specs
    • set GRPC_DNS_RESOLVER=native in the Dockerfiles of gRPC applications so they used the libc resolver
    • checked with tcpdump port 53 that A and AAAA queries left from different ports
    Lessons

    Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.

    View case
Check your cluster for this

These 9 cases involve conntrack table and CoreDNS. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

DNS or network fault9 cases

Component
conntrack table4CoreDNS2Client-side DNS resolver and cache2kube-proxy2Linux kernel2AWS VPC CNI2Cloud load balancer1Horizontal pod autoscaler1Ingress controller1Kubernetes API server1kubelet1Container memory limit1NCCL1NIC, cable and switch port link1Node operating system image1Pod termination1Container registry and its credentials1Telemetry agent1
Contributing factor
Unreviewed defaults6Unverified assumption2Ignored early signal2Automation amplification1Unpatched known bug1Unknown limit1Harmful manual action1No caching1Overlapping failures1Single point of failure1
Organisation
MindTickle2Adevinta1ByteDance1DigitalOcean1GitHub1Lotte E-commerce1Preply1Prezi1
  1. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled
    DigitalOcean2024
  2. IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUs
    ByteDance2024
  3. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS
    Adevinta2023
  4. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022
  5. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020
  6. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node
    MindTickle2020
  7. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
  8. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race
    Prezi2019
  9. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race
    MindTickle2019