Incident Library
Applied filters Componentkube-dnsSymptomHigh latencyOrganisationAirbnbClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
    Symptoms

    On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

    High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures
    Trigger

    Several services sent a high volume of external DNS queries, all of which went through kube-dns.

    Root cause
    Capacity exhaustion
    Component
    kube-dnskube-proxyClient-side DNS resolver and cache
    Contributing factor
    Unreviewed defaults
    Detection

    Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

    Lessons

    By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

    View case
Check your cluster for this

This case involves Client-side DNS resolver and cache and kube-dns. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Client-side DNS resolver and cachekube-dnskube-proxy
Contributing factor
Unreviewed defaults
Organisation
Airbnb
  1. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019