Cases5 cases

1 to 5 of 5

1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics

   UK Ministry of Justice2023

   Symptoms

   At 10:15 on 1 November 2023 a Pingdom check raised a PagerDuty high-priority alert that the "Prometheus - live healthcheck is DOWN". The same alert fired for the third time within a few minutes, and the incident was declared at 10:41.

   Monitoring outageProbe failures

   Root cause

   unknown

   Component

   Prometheus

   Contributing factor

   –

   Detection

   A Pingdom health check on the live Prometheus failed and raised a PagerDuty high-priority alert.

   Remediation

   • increased the Prometheus readiness probe timeout from 3 to 6 seconds

   Action items

   • Inspect more closely to identify this kind of failure earlier
   • Investigate whether the ingestion of data into the database is too big or too long
   • Check whether executing some queries makes Prometheus work harder and stop responding to the readiness probe
   • Check whether any other service probing Prometheus triggers the restart
   • Check whether regular Velero backups disturb EBS reads and writes and cause the restart

   View case
2. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node

   Fyra Labs2023

   Symptoms

   At around 2 AM ICT on 2023-04-09 the team saw unusual CPU use on one cluster node, which throttled every other service on it and made them fail. They presumed an upstream container runtime bug in containerd and runc and decided to fully reboot the affected nodes, taking the cluster down for maintenance on the assumption that it was not in active use.

   Sustained high node CPUService outage

   Trigger

   A full cold reboot of the nodes that showed unusual CPU use, followed by upgrades and cold reboots of the other nodes.

   Root cause

   unknown

   Component

   Multus meta-CNICalicoContainer runtime

   Contributing factor

   Harmful manual actionUnverified assumption

   Detection

   The team saw unusual CPU use on a node around 2 AM ICT and failing services on the rebooted nodes around 2:30 ICT; developers reported the Terra repositories down at 6:31 ICT.

   Remediation

   • uninstalled Multus and Calico and replaced them with the Cilium CNI
   • rolled back most cluster settings except the CNI

   Lessons

   Rebooting nodes to clear one problem can expose another; here a CNI chain could not reach the API server after a cold start. Repeating the same reboot on more nodes while the first failure was not understood spread the outage from a few nodes to the whole cluster. The team also found that Cilium's real resource use in their cluster was acceptable, contrary to the benchmark they had relied on.

   View case
3. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case
4. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC

   Binder Project2018

   Symptoms

   On 13 March 2018 at 07:28 CET the mybinder.org team noticed that JupyterHub had not been spawning new users and that the launch success rate on the Grafana dashboard had been at zero for several hours. Many pods were in the states "Unknown" and "NodeLost", on at least the nodes `10.128.0.7` and `10.128.0.5`.

   Service outageNode NotReadyPods stuck in TerminatingPods stuck in ContainerCreating

   Root cause

   unknown

   Component

   Persistent volumes and stateful setsNode pool

   Contributing factor

   Pages not delivered

   Detection

   The team noticed only after several hours, when the launch success rate on the Grafana dashboard had been at zero for a long time. No Stackdriver alert email arrived in time, because those emails were marked as spam and reached the team hours later.

   Remediation

   • Cordoned the bad nodes
   • Deleted the hub pod in Unknown state and the hub pod stuck in ContainerCreating
   • Reset node gke-prod-a-ssd-pool-32-134a959a-n2sk from the GCP web interface to force the PVC to be released
   • Force-deleted every pod in Unknown or NodeLost state with kubectl delete --grace-period=0 --force

   Action items

   • Investigate why the outage started, through Stackdriver logs and Google Cloud incidents
   • Set up an alerting service that does not depend on Stackdriver
   • Document how to check whether a PVC has not been released
   • Document how to restart a node by hand when commands are not working
   • Document how to delete pods that are in Unknown state

   Lessons

   The error went unnoticed for a long time because Stackdriver alert emails were marked as spam and arrived hours late. Once the problem was found, each step was logged and the steps were made more forceful as needed, so the hub was down only briefly after detection.

   View case
5. IL-0507 Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodes

   UC Berkeley2017

   Symptoms

   On 10 October 2017 at 10:45 AM, a post on the course Piazza reported that two students could not start their servers on UC Berkeley's DataHub, and the /hub/admin interface could not start them either. At first it was thought the students might have run out of memory.

   Pods stuck in Terminating

   Root cause

   unknown

   Component

   Container runtime

   Contributing factor

   –

   Detection

   Students reported on the course Piazza at 10:45 AM, and later in the course Slack, that their servers would not start.

   Remediation

   • Force-deleted the stuck user pods with kubectl --namespace=prod delete pod jupyter-<name> --grace-period=0 --force
   • Cordoned node k8s-pool1-19522833-13
   • Force-deleted the stuck pods on k8s-pool2-19522833-9 and cordoned that node
   • Stopped and started k8s-pool1-19522833-13 from the Azure portal, after which Docker responded, and uncordoned it
   • Stopped and started k8s-pool2-19522833-9 from the Azure portal

   Action items

   • When several student servers fail to start or stop, check whether their pods ran on the same nodes
   • Find out how many nodes do not mount /var/lib/docker on sdb1
   • Watch for elevated counts of pods stuck in Terminating

   View case

By root cause1