Cases2 cases

1 to 2 of 2

1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics

   UK Ministry of Justice2023

   Symptoms

   At 10:15 on 1 November 2023 a Pingdom check raised a PagerDuty high-priority alert that the "Prometheus - live healthcheck is DOWN". The same alert fired for the third time within a few minutes, and the incident was declared at 10:41.

   Monitoring outageProbe failures

   Root cause

   unknown

   Component

   Prometheus

   Contributing factor

   –

   Detection

   A Pingdom health check on the live Prometheus failed and raised a PagerDuty high-priority alert.

   Remediation

   • increased the Prometheus readiness probe timeout from 3 to 6 seconds

   Action items

   • Inspect more closely to identify this kind of failure earlier
   • Investigate whether the ingestion of data into the database is too big or too long
   • Check whether executing some queries makes Prometheus work harder and stop responding to the readiness probe
   • Check whether any other service probing Prometheus triggers the restart
   • Check whether regular Velero backups disturb EBS reads and writes and cause the restart

   View case
2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case

By root cause1