Cases13 cases

10 to 13 of 13

1. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop

   Fullstaq2019

   Symptoms

   Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.

   Control plane unresponsiveMonitoring outage

   Trigger

   The dex namespace was deleted so that Kubernetes would clean up the accumulated AuthRequest objects itself.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serveretcdCluster authentication provider

   Contributing factor

   Unreviewed defaultsHarmful manual actionRetry stormLimited debug accessIgnored early signal

   Detection

   The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.

   Remediation

   • removed the kubernetes finalizer from the dex namespace by calling the API directly (kill-kube-ns)
   • deployed Dex to a new namespace with sane defaults

   Lessons

   Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.

   View case
2. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances

   Datadog2019

   Symptoms

   Users reported that their jobs were not starting, with some sign that image pulls were involved. Many pods across the cluster were in error states related to image pulls, while applications that were already running were not affected.

   Image pull failuresService outage

   Trigger

   A change to the permissions on a storage bucket that one application needed to read. The application could no longer start, and its DaemonSet, running on roughly a thousand nodes of the cluster, went into a crash loop with imagePullPolicy: Always, so every restart pulled the image again although it was already on the node.

   Root cause

   Capacity exhaustion

   Component

   Container registry and its credentialsNAT gatewayApplication databaseAdmission webhook

   Contributing factor

   Automation amplificationSingle point of failureUnknown limitHarmful manual actionExcessive blast radius

   Detection

   Users reported that their jobs were not starting; monitoring had not caught it first.

   Remediation

   • replaced the affected NAT instances so registry traffic came from new source IPs
   • reconfigured the static IPs and firewall rules so applications could reach the Cloud SQL database again

   Action items

   • Add an admission webhook that rejects floating image tags, starting with latest
   • Apply the image tag webhook to workload objects only, not to pods, after it blocked pods created by in-cluster controllers

   Lessons

   The speakers' takeaway is to be very careful with DaemonSets in large clusters: they say they had broken everything DaemonSets depend on, including DNS, Vault and the image repository, and that it is very easy to break things when managing them by the thousand. imagePullPolicy: Always made every crash-loop restart pull the image again. Sending all registry traffic through three NAT instances put the whole region behind three source IPs, which the registry rate-limited, and changing those IPs cut the connections to a critical database reached over the internet through them. An admission webhook on pods rejects the pods that controllers in the cluster create, where no user sees the error.

   View case
3. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case
4. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage

   Monzo2017

   Symptoms

   At 14:10 an engineer deployed a change to a service that Monzo's current account needs to process payments, and every request to that service started to fail. Current account customers saw payment failures; the prepaid card was not affected at first because it does not use that service. The change was rolled back at 14:12, but the errors continued.

   High error rateStale endpoint routingService outage

   Trigger

   At 14:10 an engineer deployed a change to a service needed to process current account payments. Two weeks earlier the etcd cluster had been upgraded and grown from three to nine nodes, and one day earlier another team had scaled a new service down to zero replicas while leaving its Kubernetes service in place.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverLinkerdkubelet

   Contributing factor

   Ineffective rollbackHarmful manual actionOverlapping failuresMonitoring blind spotSingle point of failure

   Detection

   Payment failures began right after the 14:10 deploy. The team rolled back at 14:12 and declared an outage internally at 14:16. At 15:13, when no service could serve requests, practically all automated alerts fired.

   Remediation

   • Rolled back the payments service change (the errors continued)
   • Activated internal backup processes so most customers could still use their card
   • Restarted all linkerd instances (this made the outage worse)
   • Restarted the three Kubernetes API server processes so replacement linkerd instances could start
   • Began deploying a newer linkerd version with the fix for empty services
   • Deleted the Kubernetes service that had no endpoints, which let linkerd load service discovery again

   Action items

   • Fix the bug in Kubernetes that can trigger timeouts after a cluster reconfiguration
   • Roll out a new version of linkerd that fixes the parsing error
   • Create better health checks, dashboards and alerts for the affected components so they give clearer signals about what is wrong and prevent human error
   • Improve procedures to communicate outages internally and externally as clearly and quickly as possible

   Lessons

   A large failure in a distributed system can be very hard to understand, and well-meant human action can compound it, as restarting every linkerd instance did here by exposing a second incompatibility. Because Monzo deploys often and its network changes constantly, the deployment and request-routing systems have to be trustworthy. Monzo committed to publishing a similar post-mortem for any incident of this severity.

   View case

By root cause5