Incident Library
Applied filters Root causeunknownComponentPrometheusSymptomProbe failuresClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics
    UK Ministry of Justice2023
    Symptoms

    At 10:15 on 1 November 2023 a Pingdom check raised a PagerDuty high-priority alert that the "Prometheus - live healthcheck is DOWN". The same alert fired for the third time within a few minutes, and the incident was declared at 10:41.

    Monitoring outageProbe failures
    Root cause
    unknown
    Component
    Prometheus
    Contributing factor
    –
    Detection

    A Pingdom health check on the live Prometheus failed and raised a PagerDuty high-priority alert.

    Remediation
    • increased the Prometheus readiness probe timeout from 3 to 6 seconds
    Action items
    • Inspect more closely to identify this kind of failure earlier
    • Investigate whether the ingestion of data into the database is too big or too long
    • Check whether executing some queries makes Prometheus work harder and stop responding to the readiness probe
    • Check whether any other service probing Prometheus triggers the restart
    • Check whether regular Velero backups disturb EBS reads and writes and cause the restart
    View case
Check your cluster for this

This case involves Prometheus. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

unknown1 case

Component
Prometheus
Contributing factor
–
Organisation
UK Ministry of Justice
  1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics
    UK Ministry of Justice2023