Incident Library
Applied filters Root causeCapacity exhaustionComponentPrometheusOrganisationUK Ministry of JusticeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
    Symptoms

    At 12:32 Prometheus on the Cloud Platform was not available after a planned change, and the Prometheus pod had restarted several times. The platform lost its monitoring and had missing metrics for the period.

    Monitoring outageProbe failures
    Trigger

    A planned change restarted Prometheus at a time when the WAL count was high.

    Root cause
    Capacity exhaustion
    Component
    Prometheus
    Contributing factor
    Unreviewed defaultsUnknown limit
    Detection

    The team saw that the Prometheus pod had restarted several times after the planned change.

    Remediation
    • raised the Prometheus startupProbe threshold to 30 minutes
    Action items
    • Perform planned Prometheus restarts when the WAL count is lower
    • Set the default CPU and memory requests to meet maximum usage
    • Create a test setup that recreates the live WAL count
    • Explore the memory-snapshot-on-shutdown and auto-gomaxprocs feature flags
    • Explore storing WAL files remotely in a different location
    • Look into a blue-green Prometheus to test changes on a live-like setup before applying them to live
    • Spike into Amazon Managed Prometheus
    View case
  2. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
    Symptoms

    At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

    Monitoring outageOOMKilledNode NotReady
    Trigger

    The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode poolContainer memory limit
    Contributing factor
    No alert on limit
    Detection

    A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

    Remediation
    • drained the monitoring node, moving Prometheus to another monitoring node
    • changed the node instance size to r6i.4xlarge
    • raised the Prometheus memory limit to 100Gi
    • set the Prometheus container limits to 12 CPU cores and 110Gi of memory
    Action items
    • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)
    View case
Check your cluster for this

These 2 cases involve Prometheus and Container memory limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Prometheus2Container memory limit1Node pool1
Contributing factor
Unreviewed defaults1Unknown limit1No alert on limit1
Organisation
UK Ministry of Justice2
  1. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  2. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023