IL-0176 LINE · 2020-07-10 (published)kubernetesobservability
The cluster ran Kubernetes 1.13.5 with 235 nodes, 7 control plane nodes and 5 etcd nodes plus worker nodes, all of them VMs. Prometheus 2.14.0 ran with storage.tsdb.retention.size=25GB on a host with 4 vCPUs, 8GB of memory and a 100GB SSD. Exit code 137 means the process received SIGKILL. The author ruled out a manual kill, since only the author had root on the host, and traced through the kubelet and cAdvisor code that kubelet only reports OOM events that cAdvisor reads from /dev/kmsg; so the kernel OOM killer, not kubelet, had killed the container. Prometheus loaded more than two million series within minutes of starting, which needed more memory than the host had.
The author then looked at why the kernel chose Prometheus. The OOM killer in the kernel in use (3.10.0-957.el7) scores processes by memory use adjusted by oom_score_adj, and the Prometheus container's oom_score_adj was 999. A SystemTap script watching writes to /proc/<pid>/oom_score_adj showed no process changing it at run time. The containerd and kubelet code showed that kubelet sets the value when the container is created, from the pod's QoS class: -998 for Guaranteed, 1000 for BestEffort, and for Burstable 1000 minus 1000 times the memory request divided by node memory. In the Prometheus pod only the configmap-reload container had requests and limits (memory 50Mi and 100Mi, CPU 100m and 200m); the Prometheus container had none. The pod was therefore Burstable, and the Prometheus container got 999, which made it the first process to be killed.
The author names two causes: wrong resource requests and limits on the Prometheus pod, which gave the container a high oom_score_adj, and no advance estimate of how much memory Prometheus needs in a large cluster. Removing most of the unnecessary settings from the Prometheus ConfigMap stopped the crashes.
One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.
kubectl describe showed the Prometheus container (prom/Prometheus:v2.14.0) Waiting with Reason CrashLoopBackOff, its last state Terminated with Reason Error and Exit Code 137, while the configmap-reload sidecar was running normally. Because Prometheus had crashed on full disks before, the author checked storage first: the 30Gi PVC was only 35% used (11G of 32G). The Prometheus log showed no error; its last lines were the start of TSDB and the WAL replay ("replaying WAL, this may take awhile", "WAL segment loaded" segment=0 maxSegment=12).
The kubelet log on the host showed the readiness probe failing with "HTTP probe failed with statuscode: 503", then "Got sys oom event from cadvisor", then the liveness probe failing with "read: connection reset by peer" and a PLEG ContainerDied event. Restarting the container with docker restart reproduced it: Prometheus used almost all of the host's memory and the kernel killed it.
The Grafana dashboard for Prometheus showed the number of time series jumping to 2,162,256 within five minutes of start. A memory calculator put that at an estimated 5.7GB, on a host with 8GB of memory and swap disabled.
Symptom Container crash loopOOMKilled
Root cause Capacity exhaustion
The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.
Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.
IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshotsOnkar Raghunath Shelke · 2026same root cause · shared kubelet
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared kubelet
IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Prometheus
IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowedUK Ministry of Justice · 2024same root cause · shared Prometheus
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared kubelet
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Prometheus
IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percentTemporal · 2023same root cause · shared Prometheus
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Prometheus
IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limitZalando · 2019same root cause · shared kubelet
IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019same root cause · shared Linux kernel