IL-0044 Saifeddine Rajhi · 2024-06-14kubernetesobservability
When Prometheus restarts it rebuilds its in-memory state by replaying the write-ahead log, the on-disk record of recent samples. In this cluster the replay needed two to three times the memory Prometheus used while running, more than 50Gi against about 30Gi in steady state, so each start ended in an OOMKill before Prometheus became ready. The problem showed up while the team was upgrading the kube-prometheus-stack chart.
The author links the problem to an upstream GitHub issue open since 2020 and to two common situations: scraping too many series, and running Prometheus close to its memory limit so that any replay overhead pushes it over. Both come down to label cardinality. With Prometheus down, PromQL could not be used to find the expensive metrics, so the author ran promtool tsdb analyze inside the Prometheus pod with kubectl exec; it pointed to high-cardinality metrics such as haproxy_server_http_responses_total. Reducing that metric, giving Prometheus more memory, or moving it to a node group with more memory are the fixes the author proposes; the write-up does not say which one ended the crash loop.
After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.
The Prometheus log contained: level=info component=tsdb msg="Deletion of corrupted mmap chunk files failed, discarding chunk files completely" err="cannot handle error: iterate on on-disk chunks: out of sequence m-mapped chunk for series ref 946594555 ...". kubectl describe pod showed the container in State: Running but with Last State: Terminated and Reason: OOMKilled.
Prometheus was being killed while replaying its write-ahead log (WAL) at startup. In normal operation it used around 30Gi of memory, but the replay needed more than 50Gi. Raising the memory limit alone was not seen as a fix, because the extra memory was needed only during the replay phase.
Root cause Capacity exhaustion
The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.
Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Prometheus, Container memory limit
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit
IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Container memory limit
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Prometheus
IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023same root cause · shared Prometheus
IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Prometheus
IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit
IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020same root cause · shared Prometheus
IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit