Incident Library

IL-0323 Wikimedia Foundation · 2019-09-22database

A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only

db1075, the primary database of section s3 in eqiad, lost its RAID battery backup unit, and the failure crashed the whole host. With the primary unavailable, no writes could happen and every s3 wiki went read-only, while reads continued from the replicas. The host was power-cycled from the iDRAC, MySQL recovered InnoDB cleanly, and read_only was removed by hand. A race between pt-heartbeat, started by puppet, and MySQL not yet being fully up made the first manual puppet run fail unnoticed, so replication lag kept being reported and the outage lasted 8 minutes longer, until a second puppet run started pt-heartbeat at 19:16. Had the host not come back, the team would have had to fail over manually and promote a replica to primary.

HP hosts at Wikimedia had crashed on BBU failures before (T231638, T225391), and db1075 belongs to a batch of 6 servers, 3 of which had already had BBU problems (T233569). Replacing db1075 as primary with a host from outside that batch was the next step (T230783). The responders were 1 DBA, 2 SREs, 1 WMCS, 1 Dev and 1 Volunteer.

Severity
–
Scope
All s3 wikis were read-only for 36 minutes; reads were not affected
Users affected
–
Region
eqiad
Time to detect
2m
Time to resolve
36m

Symptoms

At 18:40 UTC db1075, the primary database of section s3, stopped answering, and all s3 wikis went read-only because no writes could reach it. Reads kept working from the replicas. At 18:42:45 Icinga reported "PROBLEM - Host db1075 is DOWN: PING CRITICAL - Packet loss = 100%".

The first error was logged by mysqld on db1115 at 18:40:38: "OpenTable: (2003) Can't connect to MySQL server on 'db1075.eqiad.wmnet' (110 "Connection timed out")". From 18:48 the s3 replicas db2105, dbstore1004 and db1095 alerted one after another with "PROBLEM - MariaDB Slave IO: s3 on db2105 is CRITICAL: CRITICAL slave_io_state Slave_IO_Running: No, Errno: 2003, Errmsg: error reconnecting to master [email protected]:3306", followed by more alerts. At 18:47 a user on #wikimedia-operations reported a warning on otrs-wiki about a high replag database lock, and that the wiki was slower than usual.

The HOST DOWN alert went only to IRC and did not page; the first SMS page went out at 18:56. A responder was connecting at 18:55:51; the team saw that the host was unresponsive with a BBU problem, and rebooted it from the iDRAC at 19:02. MySQL started InnoDB recovery at 19:03 and finished at 19:06, and read_only=ON was removed by hand at 19:07. Lag was still being reported at 19:08 although everything was up: a manual puppet run at 19:04 had failed without anyone noticing, so pt-heartbeat had not started. A second manual puppet run at 19:16 succeeded, pt-heartbeat started and the outage ended, about 8 minutes later than it could have.

Symptom Writes rejectedReplication lagHigh latency

Root cause

Root cause Hardware failure

Component DatabaseDatabase replication
Contributing factor Pages not deliveredHarmful manual action
Trigger
The RAID battery backup unit (BBU) of db1075, the s3 primary database host, failed and the whole host crashed.

Detection

Icinga reported db1075 HOST DOWN on IRC at 18:42:45, but that alert does not page; replica Slave IO alerts followed, users reported problems on #wikimedia-operations, and the first SMS page went out at 18:56. Volunteers and staff noticed the failure before the alerts did.

Remediation

Action items

Lessons

A BBU failure should not crash a whole host, yet it had happened before on HP hosts. HOST DOWN on a primary database should page instead of only alerting on IRC. Starting pt-heartbeat through puppet races with MySQL startup, and the unnoticed failure kept lag reported and made the outage 8 minutes longer.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.