Cases2 cases

1 to 2 of 2

1. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed

   UK Ministry of Justice2024

   Symptoms

   At 12:32 Prometheus on the Cloud Platform was not available after a planned change, and the Prometheus pod had restarted several times. The platform lost its monitoring and had missing metrics for the period.

   Monitoring outageProbe failures

   Trigger

   A planned change restarted Prometheus at a time when the WAL count was high.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   The team saw that the Prometheus pod had restarted several times after the planned change.

   Remediation

   • raised the Prometheus startupProbe threshold to 30 minutes

   Action items

   • Perform planned Prometheus restarts when the WAL count is lower
   • Set the default CPU and memory requests to meet maximum usage
   • Create a test setup that recreates the live WAL count
   • Explore the memory-snapshot-on-shutdown and auto-gomaxprocs feature flags
   • Explore storing WAL files remotely in a different location
   • Look into a blue-green Prometheus to test changes on a live-like setup before applying them to live
   • Spike into Amazon Managed Prometheus

   View case
2. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics

   UK Ministry of Justice2023

   Symptoms

   At 10:15 on 1 November 2023 a Pingdom check raised a PagerDuty high-priority alert that the "Prometheus - live healthcheck is DOWN". The same alert fired for the third time within a few minutes, and the incident was declared at 10:41.

   Monitoring outageProbe failures

   Root cause

   unknown

   Component

   Prometheus

   Contributing factor

   –

   Detection

   A Pingdom health check on the live Prometheus failed and raised a PagerDuty high-priority alert.

   Remediation

   • increased the Prometheus readiness probe timeout from 3 to 6 seconds

   Action items

   • Inspect more closely to identify this kind of failure earlier
   • Investigate whether the ingestion of data into the database is too big or too long
   • Check whether executing some queries makes Prometheus work harder and stop responding to the readiness probe
   • Check whether any other service probing Prometheus triggers the restart
   • Check whether regular Velero backups disturb EBS reads and writes and cause the restart

   View case

By root cause2