Cases51 cases

36 to 40 of 51

1. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case
2. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory

   UK Ministry of Justice2020

   Symptoms

   At 10:43 on 14 August 2020 two of the six ingress controller pods in the cluster were in a crash loop. Restarting those pods did not fix them.

   Container crash loop

   Root cause

   Capacity exhaustion

   Component

   Ingress controller

   Contributing factor

   Harmful manual actionSingle point of failure

   Detection

   Two of the six ingress controller pods were seen crash-looping at 10:43; the incident was declared at 11:01.

   Remediation

   • restarted the leader ingress controller pod so the other replicas could connect and get the latest nginx configuration

   View case
3. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999

   LINE2020

   Symptoms

   One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

   Container crash loopOOMKilledMonitoring outageProbe failures

   Trigger

   After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

   Root cause

   Capacity exhaustion

   Component

   PrometheuskubeletLinux kernel

   Contributing factor

   No resource requests or limitsNo capacity planning

   Detection

   The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

   Remediation

   • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing

   Action items

   • Set appropriate resource requests and limits on the Prometheus pod
   • Reduce unneeded metrics by adjusting scrape targets or scrape interval
   • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector

   Lessons

   Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

   View case
4. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets

   loveholidays2020

   Symptoms

   Shortly after a round of changes, loveholidays saw connectivity problems and raised error rates on its aggregation service. The service is a proxy between two other services of more than 100 instances each, handles over 10,000 requests per second continuously and can spike to four times that.

   Connection resetsHigh error rateIntermittent connection failures

   Trigger

   Three changes together: a change in service architecture, halving the RAM of the nodes that run the aggregation service, and sharding a downstream service, which doubled the number of connections to track.

   Root cause

   Capacity exhaustion

   Component

   conntrack tablekube-proxyNode poolHAProxy

   Contributing factor

   Unknown limitNo capacity planningNo alert on limit

   Detection

   Rising error rates and climbing connection reset errors in HAProxy, then node_exporter conntrack metrics compared with their limit.

   Remediation

   • moved the aggregation service to nodes with more RAM
   • limited the number of aggregation service pods per node

   Action items

   • Alert when conntrack table use goes above 75%, with (node_nf_conntrack_entries / on (pod) node_nf_conntrack_entries_limit / on (pod) group_right kube_pod_info) > 0.75

   Lessons

   The conntrack limit is tied to node memory, so resizing a node pool silently resizes the connection table, and on GKE the ratio did not match the commonly cited formula. A high-traffic proxy on small nodes can exhaust it, and the symptoms are connection resets and timeouts, so conntrack use needs its own metric and alert.

   View case
5. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing

   loveholidays2020

   Symptoms

   A colleague wrote in Slack that his application was taking a long time to deploy. A deployment that normally finishes in minutes had been running for about an hour: about half of the new pods were running and serving traffic, and the other half were Pending.

   Pods PendingIP address exhaustionScale-up failure

   Trigger

   The cluster reaching 256 nodes, the most its /16 pod secondary range could hold at 256 addresses per node; every further node creation failed.

   Root cause

   Capacity exhaustion

   Component

   VPC subnet or pod address rangeCluster autoscalerNode pool

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   A colleague reported in Slack that a deployment was slow; the pending pods' events and the cluster autoscaler status ConfigMap showed the exhausted IP space.

   Remediation

   • lowered the maximum pods per node on selected node pools, cutting IP consumption by about 30%
   • moved some node pools to larger instance types

   Action items

   • Lower the maximum pods per node on node pools that do not run many pods
   • Use larger instance types in some node pools

   Lessons

   In a VPC-native GKE cluster the number of nodes, not the number of pods, uses up the pod range: by default each node reserves 256 addresses, so even a /16 is full at 256 nodes. Maximum pods per node and node size are the practical levers; larger nodes pack better but take more pods down when they fail.

   View case

By root cause1