Cases6 cases

6 to 6 of 6

1. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"

   Wikimedia Foundation2019

   Symptoms

   On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

   Control plane unresponsiveService outageTLS certificate errors

   Trigger

   At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

   Root cause

   Configuration change

   Component

   Kubernetes API serveretcdConfiguration management tool

   Contributing factor

   Change with a delayed effectHarmful manual actionNo runbook

   Detection

   Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

   Remediation

   • Rebooted the Kubernetes master and the etcd cluster, which did not help
   • Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
   • Brought etcd back to full capacity through a full disaster recovery from the backup
   • Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
   • Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA

   Action items

   • Document the etcd cluster processes
   • Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
   • Make kube2proxy more resilient to API server failures
   • Audit the Puppet CA certificates so the tools project has no more than one
   • Make the Kubernetes control plane highly available

   Lessons

   Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

   View case

By root cause4