Incident Library
Applied filters Componentkube-proxySymptomDNS resolution failuresClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout
    Cisco ThousandEyes2025
    Symptoms

    On 4 September 2025 a CoreDNS pod in a Cisco ThousandEyes Kubernetes cluster was OOMKilled.

    DNS resolution failuresStale endpoint routingOOMKilled
    Trigger

    A CoreDNS pod was OOMKilled.

    Root cause
    Software defect
    Component
    CoreDNSkube-proxyconntrack tableIstio
    Contributing factor
    Unreviewed defaults
    Lessons

    The closing slide sets three behaviours side by side: Istio's extremely aggressive DNS cache refresh with one client port reused for all DNS requests, kube-proxy's lack of a cleanup mechanism for stale conntrack entries of terminated pods (fixed in Kubernetes v1.32), and conntrack's default bidirectional UDP stream timeout of 2 minutes. The speaker describes Istio as making some very complex things simple but some very simple things complex, and presents Cilium, which looks up Services in maps instead of walking the KUBE-SERVICES chain and can replace kube-proxy entirely with its eBPF data path.

    View case
  2. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020
    Symptoms

    Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

    DNS resolution failuresHigh error rateStale endpoint routing
    Trigger

    Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

    Root cause
    DNS or network fault
    Component
    CoreDNSkube-proxyconntrack table
    Contributing factor
    Automation amplificationIgnored early signalNo caching
    Detection

    Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

    Remediation
    • reverted the latest release of the affected service
    • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten
    Action items
    • Disable the autoscaler for CoreDNS
    • Run a DNS cache server in production
    • Monitor conntrack usage
    Lessons

    Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

    View case
  3. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
    Symptoms

    On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

    High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures
    Trigger

    Several services sent a high volume of external DNS queries, all of which went through kube-dns.

    Root cause
    Capacity exhaustion
    Component
    kube-dnskube-proxyClient-side DNS resolver and cache
    Contributing factor
    Unreviewed defaults
    Detection

    Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

    Lessons

    By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

    View case
Check your cluster for this

These 3 cases involve kube-proxy and conntrack table. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Software defect1 case

Component
conntrack tableCoreDNSIstiokube-proxy
Contributing factor
Unreviewed defaults
Organisation
Cisco ThousandEyes
  1. IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeout
    Cisco ThousandEyes2025

DNS or network fault1 case

Component
conntrack tableCoreDNSkube-proxy
Contributing factor
Automation amplificationIgnored early signalNo caching
Organisation
Preply
  1. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries
    Preply2020

Capacity exhaustion1 case

Component
Client-side DNS resolver and cachekube-dnskube-proxy
Contributing factor
Unreviewed defaults
Organisation
Airbnb
  1. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019