IL-0094 2i2c · 2020-10-28kubernetesgcp
2i2c runs a JupyterHub for WER on a GKE cluster. The core pods did not have appropriate resource guarantees, and there was no dedicated pool for them, so the students' pods ran on the same nodes. Each student had a memory limit of 2G and a guarantee of only 512M, the 4x overcommit 2i2c often uses, but earlier tests had shown that students almost always use just under 1G, so the overcommit should have been only 2x. The students filled the CPU and RAM of the nodes, everything was starved of resources, and the hub froze.
The team removed memory overcommit for the WER students, which was also expected to spawn another node and relieve the existing ones, and raised resource guarantees for all core pods so they could keep working on full nodes. That change restarted the pods and moved some to a new node, and things returned to normal; the issue was closed at 09:46 PM.
At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.
At 09:21 PM the node resource use made the exhaustion clear. kubectl top node showed gke-low-touch-hubs-cluster-core-pool-b7edea69-gwrg at 1944m CPU (100%) and 10432Mi memory (98%), and gke-low-touch-hubs-cluster-core-pool-b7edea69-00sc at 220m (11%) and 6151Mi (58%). The cluster had only core nodes and no separate user nodes, so the suspicion was that the student pods used just enough resources to starve the core pods.
Root cause Capacity exhaustion
WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).
Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool, Container memory limit
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Node pool, Container memory limit
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool, Container memory limit
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool, Container memory limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared Node pool
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit