Cases4 cases

1 to 4 of 4

1. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors

   MonotaRO2023

   Symptoms

   When MonotaRO raised the share of requests for the remaining endpoints of its backend API sent to EKS from 1% to 25%, Sentry filled with errors saying the API could not reach the database and outside services. The errors were about DNS name resolution, and the traffic had to be shifted back to the EC2 version.

   DNS resolution failuresHigh error rate

   Trigger

   MonotaRO raised the share of requests for the remaining API endpoints routed by the ALB to the EKS version from 1% to 25%.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSClient-side DNS resolver and cacheContainer CPU limitHorizontal pod autoscalerPod termination

   Contributing factor

   Unreviewed defaultsNo resource requests or limitsUnrepresentative stagingUnverified assumption

   Detection

   Sentry showed a large number of errors about connecting to the database and outside services right after the traffic share was raised.

   Remediation

   • Shifted the requests back to the EC2 version to stop the errors
   • Added an HPA to autoscale the CoreDNS pods as a stopgap, which let the migration to EKS finish

   Action items

   • Set ndots to 1 through dnsConfig in the application Deployments
   • Set cache 600 in the Corefile in the coredns ConfigMap to lengthen the CoreDNS cache TTL
   • Add lameduck 30s to the CoreDNS health plugin so CoreDNS keeps serving while it is scaled in
   • Consider reducing the differences between production and staging, weighed against cost

   Lessons

   MonotaRO assumed that a managed service would not cause problems and so skipped researching other companies' migrations, load testing and monitoring. Staging differed from production in its network layout, so ndots differed between them, and staging databases could not take a high-load test. Looking at the CoreDNS logs and resolv.conf instead of guessing found that ndots:5 made lookups of outside names try every search domain first.

   View case
2. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS

   Adevinta2023

   Symptoms

   On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.

   High error rateDNS resolution failuresHigh latency

   Root cause

   DNS or network fault

   Component

   CoreDNSClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unreviewed defaultsUnknown limitIgnored early signalUnverified assumption

   Detection

   A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.

   Remediation

   • raised the dnsmasq concurrent query limit from 150 to 1500
   • added a CoreDNS header block that sets the ra and aa flags and clears rd, so dnsmasq caches cluster.local answers
   • configured dnsmasq on each node to answer NXDOMAIN for unresolvable search-domain names instead of forwarding them to CoreDNS
   • changed the fluent-bit log router hostname to a fully qualified name
   • asked the affected tenants to use fully qualified names for internal calls

   Lessons

   The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated. The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did. With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.

   View case
3. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case
4. IL-0050 Pods on Kubernetes failed to resolve an external Redis hostname because of ndots:5, the musl resolver in Alpine images, and Netty's DNS resolver

   Toyota Connected2018

   Symptoms

   Services that used the team's Redis-dependent library crashed on start-up, and their teams reported it to the ops members. The Spring Boot logs showed the applications failing to resolve the Redis hostname during bean initialisation.

   DNS resolution failuresIntermittent connection failures

   Root cause

   Software defect

   Component

   Client-side DNS resolver and cache

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   Service teams reported start-up crashes to the ops members of the team; later the resolution failures were seen in Kibana.

   Remediation

   • set ndots to 4 with a dnsConfig block in the Kubernetes deployment manifests
   • corrected the maximum connection pool size in the Redis library, cutting Redis connections by about 70%

   Action items

   • Test Debian glibc-based base images added to the image library to see whether they resolve the DNS problems
   • Update dependencies to newer Netty versions

   Lessons

   With ndots:5, a name outside the cluster that has fewer than five dots is first tried with every cluster search domain. Adding a trailing dot avoids that, but only if every library in the path accepts it, and the Netty version in use did not. The ndots change is a workaround that will not hold forever. The author would like better alerting for this kind of failure but notes that alerts on one-off DNS failures are not wanted.

   View case

By root cause3