Cases2 cases

1 to 2 of 2

1. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed

   UK Ministry of Justice2024

   Symptoms

   At 12:32 Prometheus on the Cloud Platform was not available after a planned change, and the Prometheus pod had restarted several times. The platform lost its monitoring and had missing metrics for the period.

   Monitoring outageProbe failures

   Trigger

   A planned change restarted Prometheus at a time when the WAL count was high.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   The team saw that the Prometheus pod had restarted several times after the planned change.

   Remediation

   • raised the Prometheus startupProbe threshold to 30 minutes

   Action items

   • Perform planned Prometheus restarts when the WAL count is lower
   • Set the default CPU and memory requests to meet maximum usage
   • Create a test setup that recreates the live WAL count
   • Explore the memory-snapshot-on-shutdown and auto-gomaxprocs feature flags
   • Explore storing WAL files remotely in a different location
   • Look into a blue-green Prometheus to test changes on a live-like setup before applying them to live
   • Spike into Amazon Managed Prometheus

   View case
2. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady

   UK Ministry of Justice2023

   Symptoms

   At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

   Monitoring outageOOMKilledNode NotReady

   Trigger

   The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode poolContainer memory limit

   Contributing factor

   No alert on limit

   Detection

   A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

   Remediation

   • drained the monitoring node, moving Prometheus to another monitoring node
   • changed the node instance size to r6i.4xlarge
   • raised the Prometheus memory limit to 100Gi
   • set the Prometheus container limits to 12 CPU cores and 110Gi of memory

   Action items

   • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)

   View case

By root cause1