Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
    Symptoms

    At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

    Monitoring outageOOMKilledNode NotReady
    Trigger

    The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode poolContainer memory limit
    Contributing factor
    No alert on limit
    Detection

    A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

    Remediation
    • drained the monitoring node, moving Prometheus to another monitoring node
    • changed the node instance size to r6i.4xlarge
    • raised the Prometheus memory limit to 100Gi
    • set the Prometheus container limits to 12 CPU cores and 110Gi of memory
    Action items
    • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)
    View case
Check your cluster for this

This case involves Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Container memory limitNode poolPrometheus
Contributing factor
No alert on limit
Organisation
UK Ministry of Justice
  1. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023