Incident Library
Applied filters Root causeConfiguration changeFactorNo runbookOrganisationWikimedia FoundationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission
    Wikimedia Foundation2025
    Symptoms

    From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.

    Writes rejected
    Trigger

    A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.

    Root cause
    Configuration change
    Component
    Elasticsearch or OpenSearch
    Contributing factor
    Unverified assumptionNo runbookUntested backup
    Detection

    CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.

    Remediation
    • Deleted the red indices, which brought the cluster back to green
    • Restored the broken indices
    Action items
    • Re-enable the snapshot repositories in production CirrusSearch
    • Create an operations-focused OpenSearch dashboard
    • Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters
    Lessons

    Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.

    View case
  2. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
    Symptoms

    On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

    Control plane unresponsiveService outageTLS certificate errors
    Trigger

    At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

    Root cause
    Configuration change
    Component
    Kubernetes API serveretcdConfiguration management tool
    Contributing factor
    Change with a delayed effectHarmful manual actionNo runbook
    Detection

    Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

    Remediation
    • Rebooted the Kubernetes master and the etcd cluster, which did not help
    • Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
    • Brought etcd back to full capacity through a full disaster recovery from the backup
    • Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
    • Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA
    Action items
    • Document the etcd cluster processes
    • Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
    • Make kube2proxy more resilient to API server failures
    • Audit the Puppet CA certificates so the tools project has no more than one
    • Make the Kubernetes control plane highly available
    Lessons

    Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

    View case
Check your cluster for this

These 2 cases involve Configuration management tool and Elasticsearch or OpenSearch. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Configuration management tool1Elasticsearch or OpenSearch1etcd1Kubernetes API server1
Contributing factor
No runbook2Unverified assumption1Untested backup1Change with a delayed effect1Harmful manual action1
Organisation
Wikimedia Foundation2
  1. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission
    Wikimedia Foundation2025
  2. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019