Incident Library
Applied filters FactorUnknown limitOrganisationSaifeddine RajhiClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
    Symptoms

    After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.

    OOMKilledProbe failuresMonitoring outage
    Trigger

    During the kube-prometheus-stack Helm chart upgrade, Prometheus had to replay its write-ahead log on startup, and the replay needed more memory than its limit allowed.

    Root cause
    Capacity exhaustion
    Component
    PrometheusContainer memory limit
    Contributing factor
    Unknown limitNo headroom
    Detection

    The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.

    Remediation
    • ran promtool tsdb analyze inside the Prometheus pod to find high-cardinality metrics such as haproxy_server_http_responses_total
    Action items
    • Verify that the memory overhead during WAL replay stays within acceptable limits, for example 10 to 15%, and do not run Prometheus close to its memory limit
    • Optimise storage, scraping and remote write configuration and upgrade to newer Prometheus releases
    • Recover automatically from OOM crash loops, for example by deleting the WAL on OOM events
    • Enforce scrape limits so Prometheus stops scraping targets when memory use passes a threshold
    Lessons

    Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.

    View case
Check your cluster for this

This case involves Container memory limit and Prometheus. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Container memory limitPrometheus
Contributing factor
Unknown limitNo headroom
Organisation
Saifeddine Rajhi
  1. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024