IL-0099 2i2c · 2025-05-12kubernetes
2i2c runs a JupyterHub for the Earthscope community. During a workshop, users started servers from a legacy profile option that guaranteed each server 8 GB of memory on a 32 GB node but set no memory limit. When users kept using more than their guarantee, the node's memory was heavily overcommitted, users' pods evicted each other, kernels were killed and server starts took more than 15 minutes.
The first response on the evening of 12 May was a helm rollout and hub redeploy, which helped only partly; the instructor switched to a local Python environment for the rest of the day. On 13 May at 14:30 the cause was identified as the legacy profile option. The engineers did not want to change the profile options so soon before that day's workshop, so they made the 'medium' profile available so more users could share a machine, raised its guarantee so the overcommitment was smaller, and pre-warmed the cluster with 15 nodes for the 30 users. The community confirmed at 19:28 that the workshop was running smoothly, and the minimum node count was set back to zero at 19:31.
On 14 May, before the next workshop, the engineers changed the community's profile options for small nodes so that the memory guarantee equals the memory limit. The community confirmed that workshop went well.
On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.
A helm rollout and hub redeploy at 21:05 seemed to help, and the community could launch servers again, but things still hung. At 22:00 an engineer reported the problem was still there, with the error `tornado.web.HTTPError: HTTP 404: Not Found (Kernel does not exist`.
A second engineer went through the saved logs, found nothing obviously wrong with the hub, and concluded it was resource exhaustion. A Grafana panel showed high CPU usage on the core node where the hub was running.
Symptom Pod eviction
Root cause Capacity exhaustion
A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.
A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment.
For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool, Container memory limit
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool, Container memory limit
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool, Container memory limit
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool, Container memory limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared Node pool
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit