IL-0299 Wikimedia Foundation · 2025-05-28database

About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission

At 15:17 on 28 May 2025 a Data Platform SRE banned 15 elastic/opensearch hosts to prepare them for decommission, and at 21:08 merged a Puppet patch that removed access to the cluster for all the hosts it targeted, elastic/cirrussearch10[53-67]. Banning is a best-effort action and did not work as expected: some of these hosts still held primary shards, so removing them put about 60 indices into red, unwritable state. There was no maintenance plan or runbook for decommissioning this many servers of one type at once. The datacenter was depooled at the time, so users were not affected.

At 09:29 on 29 May an engineer deleted the red indices, which brought the cluster back to green, writable status and let the update pipeline recover. From 09:47 the engineer restored the broken indices, and all indices were recovered by 14:50, when the incident ended.

Severity: –
Scope: About 60 CirrusSearch OpenSearch indices red and unwritable; the search update pipeline could not write to them
Users affected: No user impact, because the datacenter was depooled
Region: –
Time to detect: 19m
Time to resolve: 17h 42m

Symptoms

From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.

Many other alerts did not fire, because they were suppressed for maintenance work on the cluster, and troubleshooting started at 07:24 the next morning. During recovery, snapshots, dashboards and logging were not available or not documented well enough.

Symptom Writes rejected

Root cause

Root cause Configuration change

Component Elasticsearch or OpenSearch

Contributing factor Unverified assumption No runbook Untested backup

Trigger: A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.

Detection

CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.

Remediation

Deleted the red indices, which brought the cluster back to green
Restored the broken indices

Action items

Re-enable the snapshot repositories in production CirrusSearch
Create an operations-focused OpenSearch dashboard
Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters

Lessons

Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.

Sources

https://wikitech.wikimedia.org/wiki/Incidents/2025-05-29_OpenSearch_clusters_unavailable · runbook · archived copy

References

https://phabricator.wikimedia.org/T388610

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Elasticsearch or OpenSearch among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0299
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0299/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.