Cases2 cases

1 to 2 of 2

1. IL-0440 Pods on Cybozu's Neco Kubernetes platform got frequent no such host and i/o timeout DNS errors from dropped packets

   Cybozu2024

   Symptoms

   A team using Neco, the Kubernetes platform run by Cybozu's Cloud Platform team, reported what looked like a performance problem in the cluster DNS servers. An application in a pod that resolved cluster Services at hundreds of requests per second frequently got `no such host` and `i/o timeout` errors.

   DNS resolution failuresIntermittent timeoutsPacket loss

   Trigger

   The number of MySQL clusters watched by the monitoring application increased, and after that the name resolution errors became frequent.

   Root cause

   Capacity exhaustion

   Component

   Linux kernelClient-side DNS resolver and cache

   Contributing factor

   Unreviewed defaults

   Detection

   A team using the platform contacted the Cloud Platform team, describing the errors as a performance problem in the cluster DNS server.

   Remediation

   • Raised the kernel parameter netdev_max_backlog on the nodes
   • Set up alerts on softnet
   • Wrote a runbook for the softnet alerts

   Action items

   • Evaluate Cilium's netkit device so pod packets bypass the backlog queue

   Lessons

   Raising netdev_max_backlog makes the same failure less likely and the new alerts catch it early, but it is not a root fix: a pod that sends more packets can fill the backlog queue again. Cybozu sees replacing the veth interface with netkit, which skips the backlog queue, as the root fix.

   View case
2. IL-0439 Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cache

   Cybozu2018

   Symptoms

   For years, at Cybozu, database access slowed down for a while each time a filesystem was expanded. Customers saw it as slower responses from the application.

   High latency

   Trigger

   Operators expanded a block device, a RAID1 multiple device made of two iSCSI devices backed by LVM volumes on remote servers, and then expanded the filesystem on it because the database had grown.

   Root cause

   Software defect

   Component

   Linux kernelDatabase

   Contributing factor

   –

   Detection

   The slowdown appeared as worse response times of the application customers use.

   Remediation

   • Patched check_disk_size_change() so the kernel invalidates the page cache only when a block device shrinks
   • Tested the patched kernel in the development environment and applied it in production

   Action items

   • Get the patch merged into the upstream Linux kernel so it need not be backported at each future OS upgrade

   Lessons

   When the intent of kernel code is unclear, the comments and the description of the commit that introduced it are the standard place to learn why it was written. The fix was 4 added lines and 2 removed lines, but reaching it took much investigation, so the effort of a source fix does not follow the number of lines changed.

   View case

By root cause2