Incident Library

IL-0440 Cybozu · 2024-08-20 (published)kubernetesnetworking

Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets

Neco runs two DNS servers: node-dns, an Unbound caching server on every node, and cluster-dns, the authoritative CoreDNS server for the cluster. A pod sends its query to node-dns, which answers from its cache or asks cluster-dns. Pod networking is mainly Cilium, mostly without an overlay network; a packet sent by a pod passes through the pod's veth interface and the Linux kernel network stack before reaching the NIC.

The monitoring application sent about 15,000 packets a second from one pod. On the way from the pod's veth interface to the host side, packets are put on a per-CPU backlog queue, whose length is set by the kernel parameter net.core.netdev_max_backlog. On the Kubernetes nodes it was the default 1000, so with this pod on the node the queue filled and outgoing packets, including DNS queries, were dropped. A lookup of the Service moco-1 sends five queries in turn because of the search domains (moco-1.default.svc.cluster.local, moco-1.svc.cluster.local, moco-1.cluster.local, moco-1.local and moco-1.). When only the first query was dropped, the others reached node-dns and got NXDOMAIN, so the application reported no such host; when all five were dropped, no answer came and the application reported i/o timeout.

Cybozu reproduced the drops in dctest, a virtual data center built from VMs that runs a Kubernetes cluster, with a Go program sending packets at a similar rate for about an hour. /proc/net/softnet_stat showed 790 and 183 dropped packets on CPU cores 8 and 9, and `perf record -e skb:kfree_skb` on the pod's cgroup gave the drop reason CPU_BACKLOG. The kernel describes that reason as a full backlog queue or the RPS flow limit, and the RPS flow limit was disabled in this environment. Cybozu raised netdev_max_backlog and added softnet alerts with a runbook. It notes that this is not a root fix, since a pod sending even more packets could fill the queue again, and plans to evaluate Cilium's netkit device (Linux kernel 6.7, Cilium 1.16), which sends pod packets straight to the physical device without passing through the backlog queue.

Severity
–
Scope
Name resolution of cluster Services from a MySQL cluster monitoring application failed intermittently
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

A team using Neco, the Kubernetes platform run by Cybozu's Cloud Platform team, reported what looked like a performance problem in the cluster DNS servers. An application in a pod that resolved cluster Services at hundreds of requests per second frequently got `no such host` and `i/o timeout` errors.

The application monitors MySQL clusters: it connects to each cluster every second, runs SQL queries to check the cluster works, and recreates its session every second. The errors became frequent after the number of MySQL clusters it monitored grew.

In the application log the failures appeared as "write check failed" with the error "ping failed: dial tcp: lookup moco-1: i/o timeout". The log of node-dns, the Unbound caching DNS server on each node, had no query from the application at the moments the application timed out; in one example about 10 seconds passed between two logged queries for moco-1.default.svc.cluster.local. The same gaps showed up in logs from other periods.

The team first suspected the DNS servers. Metrics and logs showed that node-dns and cluster-dns, the CoreDNS server in the cluster, handled their requests normally and did not return NXDOMAIN for names that existed, so the DNS queries seemed to be lost somewhere between the application and node-dns. The cadvisor metric container_network_transmit_packets_dropped_total showed about 100 packets a second dropped while being sent from the pod, and the Node exporter metric node_softnet_dropped_total, which counts packets dropped because the CPU backlog queue was full, was rising as well.

Symptom DNS resolution failuresIntermittent timeoutsPacket loss

Root cause

Root cause Capacity exhaustion

Component Linux kernelClient-side DNS resolver and cache
Contributing factor Unreviewed defaults
Trigger
The number of MySQL clusters watched by the monitoring application increased, and after that the name resolution errors became frequent.

Detection

A team using the platform contacted the Cloud Platform team, describing the errors as a performance problem in the cluster DNS server.

Remediation

Action items

Lessons

Raising netdev_max_backlog makes the same failure less likely and the new alerts catch it early, but it is not a root fix: a pod that sends more packets can fill the backlog queue again. Cybozu sees replacing the veth interface with netkit, which skips the backlog queue, as the root fix.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Linux kernel, Client-side DNS resolver and cache among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.