Cases17 cases

11 to 15 of 17

1. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes

   Grafana Labs2020

   Symptoms

   About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.

   High error rateIntermittent timeouts

   Trigger

   A scheduled Kubernetes version upgrade abruptly terminated the node that ran the etcd leader at 11:19:00 UTC.

   Root cause

   Software defect

   Component

   etcdNode pool

   Contributing factor

   Unreviewed defaultsNo safeguard on destructive change

   Detection

   An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.

   Remediation

   • Restarted the two affected Cortex distributor replicas at 11:32, which stopped the errors

   Action items

   • Enable gRPC keepalive probes in the Cortex etcd client (DialKeepAliveTime 10s, DialKeepAliveTimeout, PermitWithoutStream)
   • Make the Kubernetes upgrade process terminate pods and machines gracefully, including the etcd pods managed by the etcd operator

   Lessons

   When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer. Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.

   View case
2. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"

   Wikimedia Foundation2019

   Symptoms

   On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

   Control plane unresponsiveService outageTLS certificate errors

   Trigger

   At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

   Root cause

   Configuration change

   Component

   Kubernetes API serveretcdConfiguration management tool

   Contributing factor

   Change with a delayed effectHarmful manual actionNo runbook

   Detection

   Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

   Remediation

   • Rebooted the Kubernetes master and the etcd cluster, which did not help
   • Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
   • Brought etcd back to full capacity through a full disaster recovery from the backup
   • Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
   • Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA

   Action items

   • Document the etcd cluster processes
   • Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
   • Make kube2proxy more resilient to API server failures
   • Audit the Puppet CA certificates so the tools project has no more than one
   • Make the Kubernetes control plane highly available

   Lessons

   Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

   View case
3. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop

   Fullstaq2019

   Symptoms

   Dex, the OIDC proxy that handled Kubernetes authentication, stored its data in the Kubernetes API (storage type kubernetes, inCluster: true) and had built up a huge number of AuthRequest objects. Listing all AuthRequests timed out, and without the list they could not be deleted.

   Control plane unresponsiveMonitoring outage

   Trigger

   The dex namespace was deleted so that Kubernetes would clean up the accumulated AuthRequest objects itself.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serveretcdCluster authentication provider

   Contributing factor

   Unreviewed defaultsHarmful manual actionRetry stormLimited debug accessIgnored early signal

   Detection

   The speaker says he had got hints of the problem earlier. While the masters were down, plenty of alerts arrived in Slack, but monitoring could not be reached because authentication and port-forwarding no longer worked.

   Remediation

   • removed the kubernetes finalizer from the dex namespace by calling the API directly (kill-kube-ns)
   • deployed Dex to a new namespace with sane defaults

   Lessons

   Dex with Kubernetes as its storage keeps every AuthRequest as an API object, and with a long TTL they pile up until the API cannot list them. Deleting a namespace that holds that many objects makes Kubernetes list them in a loop and can overload the masters. Monitoring that is reached through cluster authentication is lost when authentication is what broke.

   View case
4. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API

   Civis Analytics2019

   Symptoms

   After Civis released its new script infrastructure on Kubernetes internally in September 2018, developers reported that kubectl commands were slow, and users intermittently could not start notebooks and services.

   Control plane lagControl plane unresponsiveHigh latencyHigh error rateDNS resolution failures

   Trigger

   Migrating clients' Python, R and container scripts from the in-house Bocce system to Kubernetes (the project called Croquet) pushed the cluster to about 290 EC2 instances under a large script load.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverTelemetry agentetcd

   Contributing factor

   Unknown limitUnverified assumptionMonitoring blind spotNo capacity planning

   Detection

   Developer reports of slow kubectl commands and user reports of notebooks and services failing to start, then high CPU on the master nodes.

   Remediation

   • moved internal users back to the old Bocce infrastructure
   • resized the three master nodes from m4.large to m4.4xlarge
   • changed the Datadog setup so per-node agents no longer polled the API server for service metadata
   • raised max-requests-inflight for non-mutating requests from 400 to 1200
   • grew the etcd EBS volumes from 20 to 100 GB (100 to 300 IOPS)
   • switched client scripts from the ClusterFirst DNS policy to Default

   Action items

   • Upgrade etcd from 2.2.1 to 3.x once Kops supports a safe migration
   • Solve the in-cluster DNS timeouts so client scripts can talk to other workloads in the cluster

   Lessons

   Kubernetes can reach thousands of nodes only after tuning, and third-party DaemonSets are a common cause of control plane load: an agent on every node that polls or watches the API server multiplies with the cluster size. Datadog, kube2iam and the fluentd metadata filter all showed this. A small configuration change often fixes it, but finding it took load testing and the API server access logs.

   View case
5. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe

   NRE Labs2018

   Symptoms

   The daily NRE Labs statistics showed lesson activity collapsing, and kubectl could not reach the cluster: `The connection to the server 1.2.3.4:6443 was refused - did you specify the right host or port?`

   Control plane unresponsiveProbe failuresService outage

   Trigger

   Unknown. The liveness probe that kubeadm writes into the etcd static pod manifest (an `etcdctl get foo` against 127.0.0.1:2379, failureThreshold 8, timeoutSeconds 15) started failing. The author first suspected GCP maintenance the night before, but could not reproduce the probe failure by hand and does not know why it began.

   Root cause

   unknown

   Component

   etcdkubeletKubernetes API server

   Contributing factor

   Single point of failureUnreviewed defaultsHarmful manual actionIgnored early signalMonitoring blind spot

   Detection

   The author saw lesson activity drop in the daily NRE Labs statistics during his morning routine.

   Remediation

   • removed the liveness probe section from the kubeadm-generated etcd static pod manifest and applied it between etcd restarts

   Action items

   • Add monitoring, especially for the single-node master
   • Run a basic "down for maintenance" page in Google Cloud Functions that points to Twitter for updates, to redirect to during outages

   Lessons

   Because all of the infrastructure is managed as code, the cluster could be rebuilt quickly, but destroying it before finding the cause also destroyed the evidence. Monitoring matters, especially with a single-node master. Read the restart counts in kubectl output, not just the Running status. Remember the fundamentals instead of expecting the issue to be more complicated than it is, and do not let the urgency of an outage push you into hasty decisions. Put up a maintenance page instead of leaving the site failing halfway.

   View case

By root cause8