Cases12 cases

7 to 11 of 12

1. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets

   loveholidays2020

   Symptoms

   Shortly after a round of changes, loveholidays saw connectivity problems and raised error rates on its aggregation service. The service is a proxy between two other services of more than 100 instances each, handles over 10,000 requests per second continuously and can spike to four times that.

   Connection resetsHigh error rateIntermittent connection failures

   Trigger

   Three changes together: a change in service architecture, halving the RAM of the nodes that run the aggregation service, and sharding a downstream service, which doubled the number of connections to track.

   Root cause

   Capacity exhaustion

   Component

   conntrack tablekube-proxyNode poolHAProxy

   Contributing factor

   Unknown limitNo capacity planningNo alert on limit

   Detection

   Rising error rates and climbing connection reset errors in HAProxy, then node_exporter conntrack metrics compared with their limit.

   Remediation

   • moved the aggregation service to nodes with more RAM
   • limited the number of aggregation service pods per node

   Action items

   • Alert when conntrack table use goes above 75%, with (node_nf_conntrack_entries / on (pod) node_nf_conntrack_entries_limit / on (pod) group_right kube_pod_info) > 0.75

   Lessons

   The conntrack limit is tied to node memory, so resizing a node pool silently resizes the connection table, and on GKE the ratio did not match the commonly cited formula. A high-traffic proxy on small nodes can exhaust it, and the symptoms are connection resets and timeouts, so conntrack use needs its own metric and alert.

   View case
2. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit

   Zalando2019

   Symptoms

   Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

   High error rateCredentials unavailableControl plane lag

   Trigger

   An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

   Root cause

   Capacity exhaustion

   Component

   kubeletCredential broker for podsKubernetes API serverCI system

   Contributing factor

   Unreviewed defaultsSingle point of failure

   Remediation

   • Scaled the CDP cluster up by hand so builder pods spread over more than one node

   Action items

   • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration

   View case
3. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelApplication database

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case
4. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case
5. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries

   Zalando2019

   Symptoms

   Customers of the Zalando Fashion Store got a high rate of errors on all web product pages and outfit pages for more than an hour. The errors came from the main data aggregation layer service, which runs in one of Zalando's Kubernetes clusters.

   High error rateDNS resolution failuresOOMKilledMonitoring outage

   Trigger

   Calls from the aggregation layer to a single, not very important downstream service started timing out. The aggregation layer returned 404s for them, clients retried, and the resulting spike in requests became a spike in DNS queries to CoreDNS.

   Root cause

   Capacity exhaustion

   Component

   CoreDNSContainer memory limitClient-side DNS resolver and cache

   Contributing factor

   Unknown limitNo cachingRetry stormUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   The on-call engineer for the aggregation layer phoned the on-call Kubernetes engineer. Automatic paging did not happen because the cluster's monitoring and alerting ran inside the cluster and failed along with DNS. The Kubernetes engineer then saw that the CoreDNS pods were being OOM-killed.

   Remediation

   • Raised the CoreDNS memory requests and limits by hand from 100Mi to 2000Mi

   Action items

   • Introduce a more resilient DNS setup: CoreDNS fronted by dnsmasq on every node
   • Improve high-level alerting from outside the clusters, so that a cluster problem is known before or at the same time as applications are affected
   • Improve incident response processes and tooling so that any on-call engineer can escalate directly to other teams when automatic paging fails, for example through a chat bot

   Lessons

   The Kubernetes DNS setup was far from resilient enough: a single application could take down DNS for everything in the cluster. The blast radius of one application sending more DNS queries should be limited to one node at most, ideally to single pods. Known Kubernetes DNS problems, such as Linux kernel races in conntrack handling and the ndots 5 problem, were familiar to the team and partly mitigated, but their combined effect on the DNS infrastructure was worse than the team had understood. Monitoring that runs inside the cluster was not robust enough: when the cluster's DNS broke, it could not page anyone, and recovery took longer than it should have.

   View case

By root cause1