Incident Library
Applied filters Root causeHardware failureFactorPages not deliveredOrganisationWikimedia FoundationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only
    Wikimedia Foundation2019
    Symptoms

    At 18:40 UTC db1075, the primary database of section s3, stopped answering, and all s3 wikis went read-only because no writes could reach it. Reads kept working from the replicas. At 18:42:45 Icinga reported "PROBLEM - Host db1075 is DOWN: PING CRITICAL - Packet loss = 100%".

    Writes rejectedReplication lagHigh latency
    Trigger

    The RAID battery backup unit (BBU) of db1075, the s3 primary database host, failed and the whole host crashed.

    Root cause
    Hardware failure
    Component
    DatabaseDatabase replication
    Contributing factor
    Pages not deliveredHarmful manual action
    Detection

    Icinga reported db1075 HOST DOWN on IRC at 18:42:45, but that alert does not page; replica Slave IO alerts followed, users reported problems on #wikimedia-operations, and the first SMS page went out at 18:56. Volunteers and staff noticed the failure before the alerts did.

    Remediation
    • Power-cycled db1075 from the iDRAC after finding the BBU problem
    • Let MySQL finish InnoDB recovery on db1075
    • Removed read_only=ON from db1075 by hand
    • Ran puppet by hand a second time so pt-heartbeat started
    Action items
    • Implement or refactor a script that moves replicas when the primary is unavailable
    • Fix the MediaWiki heartbeat model so pt-heartbeat does not use a super-user, avoids a single point of failure and follows the real primary without depending on puppet
    • Decide what to do with the batch of hosts that already had BBU issues
    • Buy a new BBU for db1075
    • Remove db1075 from the primary role
    • Address MediaWiki spam while the database is read-only or the primary is unavailable
    • Make primary database hosts page on HOST DOWN
    • Track hardware errors better in Netbox
    Lessons

    A BBU failure should not crash a whole host, yet it had happened before on HP hosts. HOST DOWN on a primary database should page instead of only alerting on IRC. Starting pt-heartbeat through puppet races with MySQL startup, and the unnoticed failure kept lag reported and made the outage 8 minutes longer.

    View case
Check your cluster for this

This case involves Database and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure1 case

Component
DatabaseDatabase replication
Contributing factor
Harmful manual actionPages not delivered
Organisation
Wikimedia Foundation
  1. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only
    Wikimedia Foundation2019