Cases8 cases

5 to 8 of 8

1. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case
2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case
3. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage

   Monzo2017

   Symptoms

   At 14:10 an engineer deployed a change to a service that Monzo's current account needs to process payments, and every request to that service started to fail. Current account customers saw payment failures; the prepaid card was not affected at first because it does not use that service. The change was rolled back at 14:12, but the errors continued.

   High error rateStale endpoint routingService outage

   Trigger

   At 14:10 an engineer deployed a change to a service needed to process current account payments. Two weeks earlier the etcd cluster had been upgraded and grown from three to nine nodes, and one day earlier another team had scaled a new service down to zero replicas while leaving its Kubernetes service in place.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverLinkerdkubelet

   Contributing factor

   Ineffective rollbackHarmful manual actionOverlapping failuresMonitoring blind spotSingle point of failure

   Detection

   Payment failures began right after the 14:10 deploy. The team rolled back at 14:12 and declared an outage internally at 14:16. At 15:13, when no service could serve requests, practically all automated alerts fired.

   Remediation

   • Rolled back the payments service change (the errors continued)
   • Activated internal backup processes so most customers could still use their card
   • Restarted all linkerd instances (this made the outage worse)
   • Restarted the three Kubernetes API server processes so replacement linkerd instances could start
   • Began deploying a newer linkerd version with the fix for empty services
   • Deleted the Kubernetes service that had no endpoints, which let linkerd load service discovery again

   Action items

   • Fix the bug in Kubernetes that can trigger timeouts after a cluster reconfiguration
   • Roll out a new version of linkerd that fixes the parsing error
   • Create better health checks, dashboards and alerts for the affected components so they give clearer signals about what is wrong and prevent human error
   • Improve procedures to communicate outages internally and externally as clearly and quickly as possible

   Lessons

   A large failure in a distributed system can be very hard to understand, and well-meant human action can compound it, as restarting every linkerd instance did here by exposing a second incompatibility. Because Monzo deploys often and its network changes constantly, the deployment and request-routing systems have to be trustworthy. Monzo committed to publishing a similar post-mortem for any incident of this severity.

   View case
4. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down

   Saltside2017

   Symptoms

   While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

   Pods PendingImage pull failuresNode NotReadyService outage

   Trigger

   A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

   Root cause

   Capacity exhaustion

   Component

   Node root diskkubeletContainer runtimeHelm

   Contributing factor

   Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure

   Detection

   The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

   Remediation

   • terminated the three worker nodes and let the auto scaling group replace them

   Action items

   • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
   • Increase the worker root EBS volumes from 20GB to 120GB
   • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
   • Configure different allocation limits for the different layers in Kubernetes
   • Discuss a highly available Tiller with the Helm team
   • Find a way to throttle the number of newly created pods

   Lessons

   An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

   View case

By root cause4