Incident Library
Applied filters Root causeConfiguration changeComponentDatabase replicationFactorGap in automated testsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook run
    Wikimedia Foundation2024
    Symptoms

    On 18 September 2024, while a cookbook was reconfiguring Wikimedia's core MariaDB databases from one-way eqiad -> codfw replication to circular replication ahead of a datacenter switchover, replication in the codfw -> eqiad direction broke on the x1 and es6 sections. Engineers watching the run noticed at 21:28 UTC that the x1 primary was not replicating.

    Replication lag
    Trigger

    Running the cookbook sre.switchdc.databases.prepare, which set up codfw -> eqiad replication and started pt-heartbeat, on the row-based x1 and es sections.

    Root cause
    Configuration change
    Component
    DatabaseDatabase replication
    Contributing factor
    Monitoring blind spotGap in automated testsHarmful manual action
    Detection

    Engineers watching the first run of the automation saw on orchestrator that the x1 primary was not replicating, because monitoring of circular replication was not yet enabled. The break in the eqiad -> codfw direction was reported by Icinga.

    Remediation
    • Skipped replication of the heartbeat tables with Replication_wild_ignore_table='heartbeat.%'
    • Inserted the missing heartbeat row with binary logging off on the es hosts
    • Inserted the missing heartbeat row with binary logging off on the x1 codfw hosts and removed the ignore rule
    Action items
    • Avoid the same issue again, probably by inserting the missing row when it is detected
    • Add a post-maintenance health check that replication works in both directions
    • Add options to run the maintenance in another order or on each section individually
    • Update the Phabricator task for each section rather than only at the end
    • Fix the monitoring gaps for circular replication
    • Made pt-heartbeat-wikimedia use STATEMENT format for its session on the x1 and row-based read-write es hosts, fixed in a later deploy
    Lessons

    Circular replication is not a normal working mode, so the first fix only moved the break to the other side; TRUNCATE would have worked better than DELETE because it always replicates as a statement, but it was not considered because it is more dangerous. Running sql_skip_slave_counter was avoided, since it could have made things worse or lost data. The check of the replication channel should also run once all tasks are complete, not only right after setup. The combination of row-based replication, pt-heartbeat and differing heartbeat rows was not described in earlier maintenance or documentation.

    View case
Check your cluster for this

This case involves Database and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
DatabaseDatabase replication
Contributing factor
Harmful manual actionMonitoring blind spotGap in automated tests
Organisation
Wikimedia Foundation
  1. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook run
    Wikimedia Foundation2024