Incident Library
Applied filters ComponentElasticsearch or OpenSearchSymptomWrites rejectedFactorNo runbookClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission
    Wikimedia Foundation2025
    Symptoms

    From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.

    Writes rejected
    Trigger

    A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.

    Root cause
    Configuration change
    Component
    Elasticsearch or OpenSearch
    Contributing factor
    Unverified assumptionNo runbookUntested backup
    Detection

    CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.

    Remediation
    • Deleted the red indices, which brought the cluster back to green
    • Restored the broken indices
    Action items
    • Re-enable the snapshot repositories in production CirrusSearch
    • Create an operations-focused OpenSearch dashboard
    • Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters
    Lessons

    Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.

    View case
Check your cluster for this

This case involves Elasticsearch or OpenSearch. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Elasticsearch or OpenSearch
Contributing factor
Unverified assumptionUntested backupNo runbook
Organisation
Wikimedia Foundation
  1. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission
    Wikimedia Foundation2025