Cases2 cases

1 to 2 of 2

1. IL-0299 About 60 indices of Wikimedia's CirrusSearch OpenSearch clusters went red and unwritable during a host decommission

   Wikimedia Foundation2025

   Symptoms

   From 21:08 on 28 May 2025, about 60 indices of Wikimedia's OpenSearch clusters for CirrusSearch were in red, unwritable state. At 21:27 CirrusStreamingUpdaterFlinkJobUnstable alerts began to fire because the search update pipeline could not write to those indices.

   Writes rejected

   Trigger

   A Puppet patch that removed cluster access for the hosts elastic/cirrussearch10[53-67] while some of them still held primary shards.

   Root cause

   Configuration change

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unverified assumptionNo runbookUntested backup

   Detection

   CirrusStreamingUpdaterFlinkJobUnstable alerts fired from 21:27; the review records the detection as human, by the engineer who started troubleshooting the next morning, and many other alerts were suppressed for maintenance.

   Remediation

   • Deleted the red indices, which brought the cluster back to green
   • Restored the broken indices

   Action items

   • Re-enable the snapshot repositories in production CirrusSearch
   • Create an operations-focused OpenSearch dashboard
   • Verify or fix the Logstash pipeline for Search Platform-owned OpenSearch clusters

   Lessons

   Banning nodes before removing them is a best-effort action and did not work as expected. A large operation such as decommissioning many servers at once had no maintenance plan or runbook, and the snapshots, dashboards and logging that recovery needed were not available or not documented well enough.

   View case
2. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only

   Wikimedia Foundation2019

   Symptoms

   At 18:40 UTC db1075, the primary database of section s3, stopped answering, and all s3 wikis went read-only because no writes could reach it. Reads kept working from the replicas. At 18:42:45 Icinga reported "PROBLEM - Host db1075 is DOWN: PING CRITICAL - Packet loss = 100%".

   Writes rejectedReplication lagHigh latency

   Trigger

   The RAID battery backup unit (BBU) of db1075, the s3 primary database host, failed and the whole host crashed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase replication

   Contributing factor

   Pages not deliveredHarmful manual action

   Detection

   Icinga reported db1075 HOST DOWN on IRC at 18:42:45, but that alert does not page; replica Slave IO alerts followed, users reported problems on #wikimedia-operations, and the first SMS page went out at 18:56. Volunteers and staff noticed the failure before the alerts did.

   Remediation

   • Power-cycled db1075 from the iDRAC after finding the BBU problem
   • Let MySQL finish InnoDB recovery on db1075
   • Removed read_only=ON from db1075 by hand
   • Ran puppet by hand a second time so pt-heartbeat started

   Action items

   • Implement or refactor a script that moves replicas when the primary is unavailable
   • Fix the MediaWiki heartbeat model so pt-heartbeat does not use a super-user, avoids a single point of failure and follows the real primary without depending on puppet
   • Decide what to do with the batch of hosts that already had BBU issues
   • Buy a new BBU for db1075
   • Remove db1075 from the primary role
   • Address MediaWiki spam while the database is read-only or the primary is unavailable
   • Make primary database hosts page on HOST DOWN
   • Track hardware errors better in Netbox

   Lessons

   A BBU failure should not crash a whole host, yet it had happened before on HP hosts. HOST DOWN on a primary database should page instead of only alerting on IRC. Starting pt-heartbeat through puppet races with MySQL startup, and the unnoticed failure kept lag reported and made the outage 8 minutes longer.

   View case

By root cause2