IL-0016 UK Ministry of Justice · 2023-07-25kubernetesawsobservability
Prometheus on the Cloud Platform live cluster needed more memory as the number of namespaces and Prometheus rules grew, and the instance size of its node was no longer enough to keep it running. Prometheus failed with exit code 137, its pod went into Terminating state and its node went NotReady, and node memory was later seen spiking to 89%. Draining the node only moved the problem to the next monitoring node.
The team declared the incident at 15:21, changed the instance size to r6i.4xlarge at 15:31, which doubled CPU and memory, raised the Prometheus memory limit to 100Gi at 15:50 after finding the OOMKilled pod, and at 16:18 set the container limits to 12 CPU cores and 110Gi of memory. The report gives the time to repair as 1h 50m.
At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.
When the team checked the server, Prometheus showed errors in rule evaluation and exit code 137. At 14:09 the Prometheus pod was in Terminating state, and at 14:17 the node it ran on went NotReady.
Moving Prometheus did not fix it. The team drained the monitoring node at 14:22, which moved Prometheus to another monitoring node, but at 14:56 Prometheus restarted just after coming back, and the node went back to Ready. At 15:20 the team saw node memory spiking to 89%. Even after a move to a larger instance type, Prometheus restarted again at 15:50, and the latest pod turned out to have been terminated with OOMKilled.
Root cause Capacity exhaustion
A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.
–
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit, Node pool
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit, Node pool
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Container memory limit, Node pool
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit, Prometheus
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool, Prometheus
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit, Node pool
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared Node pool
IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024same root cause · shared Prometheus
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023same root cause · shared Prometheus
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared Prometheus
IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit
IL-0001 linked by the editor