Cases12 cases

7 to 11 of 12

1. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999

   LINE2020

   Symptoms

   One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

   Container crash loopOOMKilledMonitoring outageProbe failures

   Trigger

   After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

   Root cause

   Capacity exhaustion

   Component

   PrometheuskubeletLinux kernel

   Contributing factor

   No resource requests or limitsNo capacity planning

   Detection

   The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

   Remediation

   • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing

   Action items

   • Set appropriate resource requests and limits on the Prometheus pod
   • Reduce unneeded metrics by adjusting scrape targets or scrape interval
   • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector

   Lessons

   Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

   View case
2. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case
3. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image

   Google Cloud2019

   Symptoms

   Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.

   Node kernel panic

   Trigger

   Upgrading GKE node pools to one of the affected 1.13 or 1.14 versions.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernel

   Contributing factor

   –

   Remediation

   • advised affected users to downgrade to a patch release not on the affected list, with support helping Release Channel users
   • released GKE 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12, which downgraded Container-Optimized OS to cos-73-11647-293-0
   • rolled the fixed versions out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves

   Action items

   • Ship a GKE release with an upgraded kernel that fixes the panics

   Lessons

   Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.

   View case
4. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race

   Prezi2019

   Symptoms

   Prometheus alerts fired on tail latency: from 7/24, the p99 response time of a few services went up after Prezi moved its Nginx-based reverse proxy, smartgateway, from dedicated Elastic Beanstalk machines onto shared Kubernetes worker nodes. The bump affected only the tail, so the earlier monitoring of average response times would most certainly have missed it.

   High latencyIntermittent timeouts

   Trigger

   Moving the reverse proxy onto shared EKS worker nodes, where its outgoing connections went through source NAT on the node and some SYN packets were dropped by a race in Netfilter's conntrack.

   Root cause

   DNS or network fault

   Component

   conntrack tableAWS VPC CNILinux kernelNode operating system image

   Contributing factor

   Unreviewed defaultsUnpatched known bug

   Detection

   Prometheus alerts on p99 latency fired after the reverse proxy migration.

   Remediation

   • added AWS X-Ray tracing and an $upstream_connect_time trace attribute to locate the delay
   • ran more pods of the service on more nodes, which reduced but did not end the delays
   • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true on the AWS VPC CNI to turn off SNAT on the nodes

   Lessons

   Alerting on tail latency instead of averages caught a problem that would otherwise have gone unnoticed, and tracing turned a vague slowdown into a measured connect delay. Delays of exactly 1 or 3 seconds on new outgoing connections point to dropped SYN packets, and on EKS the SNAT done by the VPC CNI is where to look.

   View case
5. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race

   MindTickle2019

   Symptoms

   Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.

   Intermittent timeoutsHigh latencyDNS resolution failures

   Trigger

   The C library resolver sent the A and AAAA queries for each lookup on the same port; where DNAT or SNAT translation takes place, a race condition in conntrack sometimes left one of the queries unanswered, and the resolver waited 5 seconds before retrying.

   Root cause

   DNS or network fault

   Component

   conntrack tableClient-side DNS resolver and cacheLinux kernel

   Contributing factor

   Unreviewed defaults

   Detection

   The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.

   Remediation

   • added the single-request-reopen option to dnsConfig in the deployment specs
   • set GRPC_DNS_RESOLVER=native in the Dockerfiles of gRPC applications so they used the libc resolver
   • checked with tcpdump port 53 that A and AAAA queries left from different ports

   Lessons

   Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.

   View case

By root cause5