IL-0070 2i2c · 2023-02-01kubernetesobservability
2i2c runs JupyterHub with dask-gateway for the LEAP community. On 1 February 2023 one user's dask-gateway workload scaled the cluster by about 100 nodes in about a minute. Prometheus ran on one of the four core nodes next to the hub and already requested most of that node's memory. The incident report's author believes that various critical pods on the core nodes were evicted when Prometheus started scraping metrics exporters on about 200 nodes. The hub pod restarted.
The notes written during the incident give two possible reasons for the hub restart: eviction under memory pressure on the node shared with Prometheus, or a hub that stopped responding, perhaps starved of CPU with a 10m request, and was restarted after its liveness probe failed 30 times in a row. The report adds that the dask scheduler pod would likely, though not certainly, also hit resource limits with that many workers. No single cause was proven, partly because the events had expired before an OOM kill could be checked.
The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.
Within about a minute roughly 100 nodes were added, probably along with 100 or more dask-worker pods. The dask workers' scheduling events read: `0/126 nodes are available: 14 node(s) didn't match Pod's node affinity/selector, 23 Insufficient cpu, 24 Insufficient memory, 6 node(s) had untolerated taint {node.kubernetes.io/network-unavailable: }, 9 node(s) had untolerated taint {hub.jupyter.org_dedicated: user}, 96 node(s) had untolerated taint {node.kubernetes.io/not-ready: }.`
The core node with the hub also ran support-prometheus-server, which requested 3 CPU and 20Gi of memory; the node had 88% of its CPU and 93% of its memory requested, and memory limits at 124%. The engineer could not confirm an OOM kill with `kubectl get event -A | grep OOM`, because the API server keeps events for only 60 minutes. Grafana showed the hub's memory steady, so a second theory was that the hub became unresponsive and its liveness probe restarted it; the hub and proxy pods requested only 10m of CPU.
Symptom Pod evictionPods Pending
Root cause Capacity exhaustion
A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.
–
Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool, Prometheus
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Node pool
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Prometheus
IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024same root cause · shared Prometheus
IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023same root cause · shared Prometheus
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared Prometheus
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool