Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
    Symptoms

    One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

    Container crash loopOOMKilledMonitoring outageProbe failures
    Trigger

    After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

    Root cause
    Capacity exhaustion
    Component
    PrometheuskubeletLinux kernel
    Contributing factor
    No resource requests or limitsNo capacity planning
    Detection

    The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

    Remediation
    • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing
    Action items
    • Set appropriate resource requests and limits on the Prometheus pod
    • Reduce unneeded metrics by adjusting scrape targets or scrape interval
    • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector
    Lessons

    Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

    View case
Check your cluster for this

These 6 cases involve In-memory cache and Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion6 cases

Component
In-memory cache3Application database3Ingress controller1kubelet1Linux kernel1Container memory limit1Node pool1Persistent volumes and stateful sets1Pod scheduling and placement1Prometheus1Telemetry agent1
Contributing factor
Harmful manual action2No capacity planning2No resource requests or limits2Retry storm2Single point of failure2Automation amplification1Excessive blast radius1Monitoring blind spot1No alert on limit1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
BC Government1DFDS1Honeycomb1LINE1PostHog1UK Ministry of Justice1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  3. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  4. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  5. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  6. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020