Incident Library
Applied filters ComponentClient-side DNS resolver and cacheSymptomPacket lossFactorUnreviewed defaultsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024
    Symptoms

    A team using Neco, the Kubernetes platform run by Cybozu's Cloud Platform team, reported what looked like a performance problem in the cluster DNS servers. An application in a pod that resolved cluster Services at hundreds of requests per second frequently got `no such host` and `i/o timeout` errors.

    DNS resolution failuresIntermittent timeoutsPacket loss
    Trigger

    The number of MySQL clusters watched by the monitoring application increased, and after that the name resolution errors became frequent.

    Root cause
    Capacity exhaustion
    Component
    Linux kernelClient-side DNS resolver and cache
    Contributing factor
    Unreviewed defaults
    Detection

    A team using the platform contacted the Cloud Platform team, describing the errors as a performance problem in the cluster DNS server.

    Remediation
    • Raised the kernel parameter netdev_max_backlog on the nodes
    • Set up alerts on softnet
    • Wrote a runbook for the softnet alerts
    Action items
    • Evaluate Cilium's netkit device so pod packets bypass the backlog queue
    Lessons

    Raising netdev_max_backlog makes the same failure less likely and the new alerts catch it early, but it is not a root fix: a pod that sends more packets can fill the backlog queue again. Cybozu sees replacing the veth interface with netkit, which skips the backlog queue, as the root fix.

    View case
Check your cluster for this

This case involves Client-side DNS resolver and cache and Linux kernel. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Client-side DNS resolver and cacheLinux kernel
Contributing factor
Unreviewed defaults
Organisation
Cybozu
  1. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets
    Cybozu2024