IL-0033 Airbnb · 2019-11-21 (published)kubernetesawsnetworking
Airbnb ran Kubernetes on AWS EC2 with Ubuntu images and the Canal CNI plugin (Calico with flannel), and was moving services that used to run on their own EC2 instances onto Kubernetes and containers; by September 2019 about a thousand services ran on Kubernetes. In this talk two Airbnb engineers go through latency problems that showed up during the migration.
Pods use the dnsPolicy ClusterFirst by default, so every DNS query, including queries for names outside the cluster, is sent to kube-dns through its cluster IP. That cluster IP is implemented with iptables rules, which pick a kube-dns pod at random rather than balancing load, so some kube-dns pods received far more queries than others. The team then hit a DNS packets-per-second limit, and DNS responses slowed down and timed out. The affected services all relied heavily on DNS and made many external lookups. Service latency rose from around one second to five and then 20 seconds across several services.
The speakers' answer is to choose the DNS policy per pod: a pod that does not need Kubernetes DNS resolution can use dnsPolicy Default, or None with its own DNS parameters, so that its external lookups do not pass through kube-dns.
On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.
The affected services had one thing in common: they relied heavily on DNS and sent many DNS queries for names outside the cluster. The query load on kube-dns was uneven: some kube-dns pods were handling a high rate of queries per second while others handled very little. DNS answers became slow and timed out for several services.
Symptom High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures
Root cause Capacity exhaustion
Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.
–
–
By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often.
iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.
IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookupsunknown · 2025same root cause · shared Client-side DNS resolver and cache
IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolutionMercari · 2025 · sev1same root cause · shared kube-dns
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Client-side DNS resolver and cache
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared kube-proxy
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Client-side DNS resolver and cache