Cases6 cases

5 to 6 of 6

1. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelApplication database

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case
2. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver

   Toyota Connected2018

   Symptoms

   Services that used the team's Redis-dependent library crashed on start-up, and their teams reported it to the ops members. The Spring Boot logs showed the applications failing to resolve the Redis hostname during bean initialisation.

   DNS resolution failuresIntermittent connection failures

   Root cause

   Software defect

   Component

   Client-side DNS resolver and cache

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   Service teams reported start-up crashes to the ops members of the team; later the resolution failures were seen in Kibana.

   Remediation

   • set ndots to 4 with a dnsConfig block in the Kubernetes deployment manifests
   • corrected the maximum connection pool size in the Redis library, cutting Redis connections by about 70%

   Action items

   • Test Debian glibc-based base images added to the image library to see whether they resolve the DNS problems
   • Update dependencies to newer Netty versions

   Lessons

   With ndots:5, a name outside the cluster that has fewer than five dots is first tried with every cluster search domain. Adding a trailing dot avoids that, but only if every library in the path accepts it, and the Netty version in use did not. The ndots change is a workaround that will not hold forever. The author would like better alerting for this kind of failure but notes that alerts on one-off DNS failures are not wanted.

   View case

By root cause4