Cases19 cases

12 to 16 of 19

1. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady

   UK Ministry of Justice2023

   Symptoms

   At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

   Monitoring outageOOMKilledNode NotReady

   Trigger

   The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode poolContainer memory limit

   Contributing factor

   No alert on limit

   Detection

   A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

   Remediation

   • drained the monitoring node, moving Prometheus to another monitoring node
   • changed the node instance size to r6i.4xlarge
   • raised the Prometheus memory limit to 100Gi
   • set the Prometheus container limits to 12 CPU cores and 110Gi of memory

   Action items

   • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)

   View case
2. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up

   2i2c2022

   Symptoms

   On 31 October 2022 at 6:27 PM Pacific time, users of the UToronto JupyterHub run by 2i2c could no longer log in or start their servers, and the hub answered with 500 errors. Users who were already logged in could keep using the hub.

   High error ratePersistent volume full

   Trigger

   The JupyterHub log file, written to the same 10G disk as the hub's sqlite database, grew until the disk was completely full.

   Root cause

   Capacity exhaustion

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No alert on limit

   Detection

   A user emailed 2i2c support at 7:13 PM that users could not log in; the team acknowledged the issue at 8:33 PM. There was no alert for the problem.

   Remediation

   • Compressed the hub log file into /tmp from a shell in the hub pod, because it could not be compressed in place on the full disk
   • Copied the compressed log to a local machine so the hub pod could be restarted without losing it
   • Resized the hub database disk (the hub-db-dir PVC) from 10G to 60G
   • Restarted the hub pod, which came back with the larger disk and restored service

   Action items

   • Stop keeping hub logs on the same disk as the hub database
   • Add a Grafana graph of free space on the JupyterHub database disk
   • Alert on elevated error rates

   Lessons

   Kubernetes dynamic volume resizing let the hub database disk grow from 10G to 60G without much effort. The risk of keeping logs on the database disk had been identified during the earlier migration, but the team had no solid process for following it through, and because no alert existed the team had to wait for a person to report the outage.

   View case
3. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired

   dbhang2021

   Symptoms

   KubeSpawner, which starts JupyterHub user containers on Kubernetes, failed to create containers with an error the author had never seen, saying an SSL certificate had expired.

   Control plane unresponsiveDeploys failing

   Root cause

   Certificate or credential expiry

   Component

   Kubernetes API serverkubelet

   Contributing factor

   Unknown limitNo alert on limit

   Detection

   KubeSpawner failed to create a container and logged an SSL certificate verification error.

   Remediation

   • checked certificate expiry with kubeadm alpha certs check-expiration
   • moved the old certificates and /etc/kubernetes/*.conf files to a backup directory
   • regenerated all certificates with kubeadm init phase certs all
   • regenerated the kubeconfig files with kubeadm init phase kubeconfig all
   • copied /etc/kubernetes/admin.conf to ~/.kube/config and restarted kubelet
   • restarted the kube-system containers with docker restart so they picked up the new credentials

   Lessons

   kubeadm clusters run on certificates that expire, and when the API server certificate expires kubectl cannot be used to fix it, so renewal has to happen on the control plane node with kubeadm. After renewal, the kubeconfig files and the running control plane containers must also pick up the new certificates, or kubectl returns Unauthorized.

   View case
4. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets

   loveholidays2020

   Symptoms

   Shortly after a round of changes, loveholidays saw connectivity problems and raised error rates on its aggregation service. The service is a proxy between two other services of more than 100 instances each, handles over 10,000 requests per second continuously and can spike to four times that.

   Connection resetsHigh error rateIntermittent connection failures

   Trigger

   Three changes together: a change in service architecture, halving the RAM of the nodes that run the aggregation service, and sharding a downstream service, which doubled the number of connections to track.

   Root cause

   Capacity exhaustion

   Component

   conntrack tablekube-proxyNode poolHAProxy

   Contributing factor

   Unknown limitNo capacity planningNo alert on limit

   Detection

   Rising error rates and climbing connection reset errors in HAProxy, then node_exporter conntrack metrics compared with their limit.

   Remediation

   • moved the aggregation service to nodes with more RAM
   • limited the number of aggregation service pods per node

   Action items

   • Alert when conntrack table use goes above 75%, with (node_nf_conntrack_entries / on (pod) node_nf_conntrack_entries_limit / on (pod) group_right kube_pod_info) > 0.75

   Lessons

   The conntrack limit is tied to node memory, so resizing a node pool silently resizes the connection table, and on GKE the ratio did not match the commonly cited formula. A high-traffic proxy on small nodes can exhaust it, and the symptoms are connection resets and timeouts, so conntrack use needs its own metric and alert.

   View case
5. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours

   Kubernetes project2019

   Symptoms

   Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

   Container crash loopService outage

   Trigger

   At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

   Root cause

   Faulty deployment

   Component

   CI systemNode pool

   Contributing factor

   Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours

   Detection

   Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

   Remediation

   • Built and deployed a new Boskos image with the bug fixed
   • Cleaned up the GCP projects that had been left open

   Action items

   • Change the Boskos configuration so that deployments actually take effect (#11956)
   • Automate Boskos deploys (#11957)
   • Improve alerting on Boskos status (#11953)

   Lessons

   The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

   View case

By root cause6