Cases3 cases

1 to 3 of 3

1. IL-0083 Intermittent 5xx errors in a multi-tenant Kubernetes cluster came from internal DNS: the node-local dnsmasq hit its 150 concurrent query limit, did not cache cluster.local answers, and ndots search lookups flooded CoreDNS

   Adevinta2023

   Symptoms

   On a Friday afternoon a tenant of SCHIP, Adevinta's multi-tenant Kubernetes runtime, reported a spike of HTTP 5xx errors in some of its services. The drops in quality of service came back periodically through the following days, each one short, and were reported mostly around midnight.

   High error rateDNS resolution failuresHigh latency

   Root cause

   DNS or network fault

   Component

   CoreDNSClient-side DNS resolver and cacheTelemetry agent

   Contributing factor

   Unreviewed defaultsUnknown limitIgnored early signalUnverified assumption

   Detection

   A tenant reported a spike of HTTP 5xx errors in some of its services; the DNS Latency SLI had already been degraded for some time without being acted on.

   Remediation

   • raised the dnsmasq concurrent query limit from 150 to 1500
   • added a CoreDNS header block that sets the ra and aa flags and clears rd, so dnsmasq caches cluster.local answers
   • configured dnsmasq on each node to answer NXDOMAIN for unresolvable search-domain names instead of forwarding them to CoreDNS
   • changed the fluent-bit log router hostname to a fully qualified name
   • asked the affected tenants to use fully qualified names for internal calls

   Lessons

   The DNS Latency SLI had shown the problem long before anyone looked at it. An SLO only protects you when its SLIs are trusted and a drop is investigated. The investigation took a month partly because several on-call groups worked on it separately, without one place for observations and hypotheses, and sometimes chased hypotheses already discarded. Disprove one hypothesis before starting the next, and collect data the metrics system lacks: DNStop showed the garbage lookups that no dashboard did. With ndots:5, short service names multiply DNS traffic; a node-local cache only helps if it can cache the cluster DNS answers at all.

   View case
2. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes

   Palark2023

   Symptoms

   One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.

   Control plane unresponsiveDatabase overload

   Trigger

   An ordinary restart of kube-apiserver, after which every pod of the log collector DaemonSet sent its list requests at the same time.

   Root cause

   Capacity exhaustion

   Component

   etcdKubernetes API serverTelemetry agent

   Contributing factor

   Unknown limitUnreviewed defaults

   Detection

   Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.

   Remediation

   • created a PriorityLevelConfiguration named limit-list-custom that queues requests (assuredConcurrencyShares 5, 16 queues, handSize 4, queueLengthLimit 50)
   • created a FlowSchema that routes list and get requests on pods from the log collector's service account to that priority level

   Action items

   • Submit a pull request adding resource version support to the Kubernetes Rust library
   • Open a pull request to the Vector log collector to address the startup list requests
   • Apply the same flow-control approach to a similar problem found with the Cilium CNI

   Lessons

   A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.

   View case
3. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case

By root cause2