Cases23 cases

16 to 20 of 23

1. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

   Kubernetes project2019

   Symptoms

   On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

   Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

   Trigger

   A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageNode root diskkubeletCI system

   Contributing factor

   Monitoring blind spotIgnored early signal

   Detection

   Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

   Remediation

   • Requested and received additional disk quota
   • Brought up 55 nodes with 500 GB disks in a new node pool
   • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55

   Action items

   • Create or fix monitoring so the oncall is paged when tide stops doing things
   • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support

   Lessons

   Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

   View case
2. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated

   UC Berkeley2019

   Symptoms

   On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

   Control plane unresponsiveService outage

   Root cause

   Cloud provider outage

   Component

   AKS control planeKubernetes API serverkubelet

   Contributing factor

   Limited debug access

   Detection

   Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

   Remediation

   • Created a new cluster and requested a new NFS server in its resource group
   • Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
   • Imported the ZFS pool into the new NFS server
   • Created the staging hub with helm and updated DNS
   • Created the prod hub with helm and updated DNS

   Action items

   • Monitor the API server endpoint remotely and alert when it is down

   View case
3. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case
4. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case
5. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures

   AirMap2018

   Symptoms

   At about 15:00 UTC on 19 July 2018, monitoring alerted on rising 5xx responses from several APIs, most notably pilot and tiles. Logins failed first, because the pilot service fetches the user's profile, and from there all services became unresponsive. Every affected API ran on the production Kubernetes cluster.

   High error rateNode NotReadyDNS resolution failuresContainer crash loopService outage

   Root cause

   Software defect

   Component

   kubelet

   Contributing factor

   Unpatched known bug

   Detection

   Monitoring alerted on an increase in 5xx responses from several APIs, most notably pilot and tiles.

   Remediation

   • restarted the kubelets on the stuck worker nodes, which did not last
   • restarted all worker nodes in a staggered order starting at about 18:00 UTC, which took about 1.5 hours

   Action items

   • Restart nodes on a rolling schedule to keep nodes and kubelets healthy until the fixes land upstream and reach the cloud provider, Microsoft Azure

   Lessons

   When nodes keep falling back to NotReady after kubelet restarts and pods fail on network and DNS errors, a known upstream bug may be the cause; matching the symptoms to the upstream issue tracker, with help from the community, found the cause and the fix candidates.

   View case

By root cause8