Incident Library

IL-0458 Wikimedia Foundation · 2019-09-10kubernetes

Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"

Puppet changes made the night before left the Toolforge Kubernetes API server with a CA certificate that did not match the one used by the etcd servers. Nothing broke until 18:30 on 10 September 2019, when maintain-kubeusers restarted the API server and it had to reconnect to etcd. The API server failed to start, kube2proxy lost its watch on Kubernetes and crashed, and Kubernetes webservices on Toolforge went offline at the proxy for several hours.

Because the change and the failure were hours apart, the responders first looked at etcd. Quirks of the old etcd cluster, such as etcdctl timing out after 1 second and frequent raft elections, sent the team down that path; the etcd cluster was rebooted, reduced to one node and, after a mistake while adding a node back to the two-node cluster, collapsed and was recovered from a backup. Around 01:00 the team turned back to the API server, found that the CA certificate had changed, and reproduced a CA trust error with curl.

At 01:28 swapping the API server's CA for the one from the etcd servers restored communication and services, but Puppet reverted the change and broke things again at 01:33. Following a pointer to an earlier Phabricator task, the etcd Puppet CA certificate was placed in /var/lib/puppet/client/ssl/certs/ca.pem, which Puppet picked up, and the outage was over at 01:43. Four SREs, a team manager and a technical volunteer took part.

Severity
–
Scope
Toolforge Kubernetes API server down for several hours: tools could not be launched or managed with webservice or kubectl, and Kubernetes webservices were offline at the proxy level after kube2proxy crashed
Users affected
Toolforge users and the tools running on the Kubernetes backend
Region
–
Time to detect
15m
Time to resolve
7h 13m

Symptoms

On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

The API server log showed the failure as "Failed to list *storage.StorageClass: client: etcd cluster is unavailable or misconfigured". kube2proxy could not watch for Kubernetes changes, started from scratch and crashed. Prometheus showed the tool containers themselves still running, so the webservices on Kubernetes appear to have gone offline at the proxy level. The Toolforge homepage alerted as down at 19:48.

Puppet certificates were suspected within the first half hour but set aside, because the Puppet changes had been made the night before and the etcd and Kubernetes certificates turned out not to have changed. The team then worked on etcd for hours: an etcd health check reported success, the same error could be reproduced against etcd, the etcd logs showed timeouts and frequent raft elections, and the old etcdctl on the tools-k8s-etcd servers timed out after 1 second, which made etcd look as if it were flapping when it was only slower than that timeout. The etcd cluster was rebooted, cut down to one node and, after a mistake while adding a node back, collapsed and had to be restored from a backup.

At 01:02 the responders noticed that the CA certificate had changed the night before in a way that was hard to pin down, and at 01:05 curl reproduced a CA trust error on SSL calls from the API server that did not occur between the etcd servers.

Symptom Control plane unresponsiveService outageTLS certificate errors

Root cause

Root cause Configuration change

Component Kubernetes API serveretcdConfiguration management tool
Contributing factor Change with a delayed effectHarmful manual actionNo runbook
Trigger
At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

Detection

Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

Remediation

Action items

Lessons

Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Kubernetes API server, etcd, Configuration management tool among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.