Cases1 case

1 to 1 of 1

1. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets

   Cybozu2024

   Symptoms

   A team using Neco, the Kubernetes platform run by Cybozu's Cloud Platform team, reported what looked like a performance problem in the cluster DNS servers. An application in a pod that resolved cluster Services at hundreds of requests per second frequently got `no such host` and `i/o timeout` errors.

   DNS resolution failuresIntermittent timeoutsPacket loss

   Trigger

   The number of MySQL clusters watched by the monitoring application increased, and after that the name resolution errors became frequent.

   Root cause

   Capacity exhaustion

   Component

   Linux kernelClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   A team using the platform contacted the Cloud Platform team, describing the errors as a performance problem in the cluster DNS server.

   Remediation

   • Raised the kernel parameter netdev_max_backlog on the nodes
   • Set up alerts on softnet
   • Wrote a runbook for the softnet alerts

   Action items

   • Evaluate Cilium's netkit device so pod packets bypass the backlog queue

   Lessons

   Raising netdev_max_backlog makes the same failure less likely and the new alerts catch it early, but it is not a root fix: a pod that sends more packets can fill the backlog queue again. Cybozu sees replacing the veth interface with netkit, which skips the backlog queue, as the root fix.

   View case

By root cause1