Incident Library
Applied filters SymptomContainer crash loopFactorNo capacity planningOrganisationLINE+Clear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
    Symptoms

    One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

    Container crash loopOOMKilledMonitoring outageProbe failures
    Trigger

    After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

    Root cause
    Capacity exhaustion
    Component
    PrometheuskubeletLinux kernel
    Contributing factor
    No resource requests or limitsNo capacity planning
    Detection

    The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

    Remediation
    • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing
    Action items
    • Set appropriate resource requests and limits on the Prometheus pod
    • Reduce unneeded metrics by adjusting scrape targets or scrape interval
    • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector
    Lessons

    Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

    View case
Check your cluster for this

This case involves kubelet and Linux kernel. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
kubeletLinux kernelPrometheus
Contributing factor
No capacity planningNo resource requests or limits
Organisation
LINE
  1. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020