Cases2 cases

1 to 2 of 2

1. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up

   2i2c2023

   Symptoms

   The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

   Pod evictionPods Pending

   Trigger

   A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode pool

   Contributing factor

   No headroomExcessive blast radiusMonitoring blind spot

   Detection

   A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

   Action items

   • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
   • Make the Grafana OOMKiller dashboard work (#2213)
   • Move LEAP cloud infrastructure to shared nodes (#2209)
   • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
   • Decide whether Prometheus should scrape dask worker pods (#2366)

   Lessons

   Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

   View case
2. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing

   loveholidays2020

   Symptoms

   A colleague wrote in Slack that his application was taking a long time to deploy. A deployment that normally finishes in minutes had been running for about an hour: about half of the new pods were running and serving traffic, and the other half were Pending.

   Pods PendingIP address exhaustionScale-up failure

   Trigger

   The cluster reaching 256 nodes, the most its /16 pod secondary range could hold at 256 addresses per node; every further node creation failed.

   Root cause

   Capacity exhaustion

   Component

   VPC subnet or pod address rangeCluster autoscalerNode pool

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   A colleague reported in Slack that a deployment was slow; the pending pods' events and the cluster autoscaler status ConfigMap showed the exhausted IP space.

   Remediation

   • lowered the maximum pods per node on selected node pools, cutting IP consumption by about 30%
   • moved some node pools to larger instance types

   Action items

   • Lower the maximum pods per node on node pools that do not run many pods
   • Use larger instance types in some node pools

   Lessons

   In a VPC-native GKE cluster the number of nodes, not the number of pods, uses up the pod range: by default each node reserves 256 addresses, so even a /16 is full at 256 nodes. Maximum pods per node and node size are the practical levers; larger nodes pack better but take more pods down when they fail.

   View case

By root cause1