Cases8 cases

7 to 8 of 8

1. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver

   Toyota Connected2018

   Symptoms

   Services that used the team's Redis-dependent library crashed on start-up, and their teams reported it to the ops members. The Spring Boot logs showed the applications failing to resolve the Redis hostname during bean initialisation.

   DNS resolution failuresIntermittent connection failures

   Root cause

   Software defect

   Component

   Client-side DNS resolver and cache

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   Service teams reported start-up crashes to the ops members of the team; later the resolution failures were seen in Kibana.

   Remediation

   • set ndots to 4 with a dnsConfig block in the Kubernetes deployment manifests
   • corrected the maximum connection pool size in the Redis library, cutting Redis connections by about 70%

   Action items

   • Test Debian glibc-based base images added to the image library to see whether they resolve the DNS problems
   • Update dependencies to newer Netty versions

   Lessons

   With ndots:5, a name outside the cluster that has fewer than five dots is first tried with every cluster search domain. Adding a trailing dot avoids that, but only if every library in the path accepts it, and the Netty version in use did not. The ndots change is a workaround that will not hold forever. The author would like better alerting for this kind of failure but notes that alerts on one-off DNS failures are not wanted.

   View case
2. IL-0051 A third-party log shipping client leaked file descriptors with every new container until nodes failed with too many open files

   thredUP2018

   Symptoms

   While thredUP's move to Kubernetes seemed to be going well, Datadog showed the number of open files on the nodes climbing steadily, with no obvious reason.

   Service outageFile descriptor exhaustion

   Trigger

   Containers starting on a node, which in Kubernetes happens constantly with every deploy and every pod that moves between nodes.

   Root cause

   Software defect

   Component

   Telemetry agent

   Contributing factor

   Unverified assumption

   Detection

   Datadog showed the number of open files on the nodes rising.

   Remediation

   • killed the affected node, brought up a new one and redeployed

   Action items

   • Move away from the log aggregation vendor

   Lessons

   Libraries and agents that worked with the EC2 deployment lifecycle may not cope with Kubernetes, where containers start, stop and move between nodes constantly. Each one needs to be checked against the new lifecycle instead of being trusted because it worked before.

   View case

By root cause1