Cases2 cases

1 to 2 of 2

1. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution

   Mercari2025sev1

   Symptoms

   During peak hours, when Mercari's GKE clusters handle very high request rates, the platform team saw kube-dns reaching its limits. DNS query rates were high enough to overwhelm the kube-dns service.

   DNS resolution failuresIntermittent timeoutsService outage

   Trigger

   Peak-hour traffic on the GKE clusters produced more DNS queries than the few kube-dns pods could answer.

   Root cause

   Capacity exhaustion

   Component

   kube-dns

   Contributing factor

   No cachingSingle point of failure

   Detection

   During peak hours the team saw frequent NXDOMAIN and truncated-response errors and recurring "failed to refresh DNS cache" errors on the clusters, which showed that kube-dns had reached its limits.

   Remediation

   • deployed NodeLocal DNSCache on the Laboratory GKE cluster with a specific nodeAffinity to measure its effect
   • rolled NodeLocal DNSCache out gradually to the Dev and Prod environments by labelling node pools so its pods could be deployed there

   Lessons

   When every pod in a busy cluster resolves names through a few central kube-dns pods, DNS becomes a shared bottleneck, and a DNS failure quickly turns into higher latency, timeouts and cascading outages across microservices. A cache on each node takes most of that load off kube-dns and keeps resolution local to the node. Rolling the cache out first to an internal test cluster with nodeAffinity, and then node pool by node pool through labels, kept the blast radius of the change small.

   View case
2. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds

   Airbnb2019

   Symptoms

   On May 10 the latency graphs of several services showed a few spikes of up to about one second, then latency climbed to five seconds and then to 20 seconds, and it stayed there for a long time. The same pattern appeared on many services, not just one.

   High latencyIntermittent timeoutsLoad imbalanceDNS resolution failures

   Trigger

   Several services sent a high volume of external DNS queries, all of which went through kube-dns.

   Root cause

   Capacity exhaustion

   Component

   kube-dnskube-proxyClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   Latency graphs of several services showed the rise; engineers then looked at load per kube-dns pod.

   Lessons

   By default a pod discovers DNS through the kube-dns cluster IP and uses the ClusterFirst policy, so even external lookups go through kube-dns. If a pod does not need cluster DNS, set its dnsPolicy to Default or None. Check the DNS policy early and often. iptables picks backends at random and is not a load balancer, so pods behind a cluster IP, kube-dns included, can carry very different loads.

   View case

By root cause1