Cases3 cases

1 to 3 of 3

1. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook run

   Wikimedia Foundation2024

   Symptoms

   On 18 September 2024, while a cookbook was reconfiguring Wikimedia's core MariaDB databases from one-way eqiad -> codfw replication to circular replication ahead of a datacenter switchover, replication in the codfw -> eqiad direction broke on the x1 and es6 sections. Engineers watching the run noticed at 21:28 UTC that the x1 primary was not replicating.

   Replication lag

   Trigger

   Running the cookbook sre.switchdc.databases.prepare, which set up codfw -> eqiad replication and started pt-heartbeat, on the row-based x1 and es sections.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replication

   Contributing factor

   Monitoring blind spotGap in automated testsHarmful manual action

   Detection

   Engineers watching the first run of the automation saw on orchestrator that the x1 primary was not replicating, because monitoring of circular replication was not yet enabled. The break in the eqiad -> codfw direction was reported by Icinga.

   Remediation

   • Skipped replication of the heartbeat tables with Replication_wild_ignore_table='heartbeat.%'
   • Inserted the missing heartbeat row with binary logging off on the es hosts
   • Inserted the missing heartbeat row with binary logging off on the x1 codfw hosts and removed the ignore rule

   Action items

   • Avoid the same issue again, probably by inserting the missing row when it is detected
   • Add a post-maintenance health check that replication works in both directions
   • Add options to run the maintenance in another order or on each section individually
   • Update the Phabricator task for each section rather than only at the end
   • Fix the monitoring gaps for circular replication
   • Made pt-heartbeat-wikimedia use STATEMENT format for its session on the x1 and row-based read-write es hosts, fixed in a later deploy

   Lessons

   Circular replication is not a normal working mode, so the first fix only moved the break to the other side; TRUNCATE would have worked better than DELETE because it always replicates as a statement, but it was not considered because it is more dangerous. Running sql_skip_slave_counter was avoided, since it could have made things worse or lost data. The check of the replication channel should also run once all tasks are complete, not only right after setup. The combination of row-based replication, pt-heartbeat and differing heartbeat rows was not described in earlier maintenance or documentation.

   View case
2. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only

   Wikimedia Foundation2019

   Symptoms

   At 18:40 UTC db1075, the primary database of section s3, stopped answering, and all s3 wikis went read-only because no writes could reach it. Reads kept working from the replicas. At 18:42:45 Icinga reported "PROBLEM - Host db1075 is DOWN: PING CRITICAL - Packet loss = 100%".

   Writes rejectedReplication lagHigh latency

   Trigger

   The RAID battery backup unit (BBU) of db1075, the s3 primary database host, failed and the whole host crashed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase replication

   Contributing factor

   Pages not deliveredHarmful manual action

   Detection

   Icinga reported db1075 HOST DOWN on IRC at 18:42:45, but that alert does not page; replica Slave IO alerts followed, users reported problems on #wikimedia-operations, and the first SMS page went out at 18:56. Volunteers and staff noticed the failure before the alerts did.

   Remediation

   • Power-cycled db1075 from the iDRAC after finding the BBU problem
   • Let MySQL finish InnoDB recovery on db1075
   • Removed read_only=ON from db1075 by hand
   • Ran puppet by hand a second time so pt-heartbeat started

   Action items

   • Implement or refactor a script that moves replicas when the primary is unavailable
   • Fix the MediaWiki heartbeat model so pt-heartbeat does not use a super-user, avoids a single point of failure and follows the real primary without depending on puppet
   • Decide what to do with the batch of hosts that already had BBU issues
   • Buy a new BBU for db1075
   • Remove db1075 from the primary role
   • Address MediaWiki spam while the database is read-only or the primary is unavailable
   • Make primary database hosts page on HOST DOWN
   • Track hardware errors better in Netbox

   Lessons

   A BBU failure should not crash a whole host, yet it had happened before on HP hosts. HOST DOWN on a primary database should page instead of only alerting on IRC. Starting pt-heartbeat through puppet races with MySQL startup, and the unnoticed failure kept lag reported and made the outage 8 minutes longer.

   View case
3. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"

   Wikimedia Foundation2019

   Symptoms

   On 10 September 2019 the Kubernetes API server of Wikimedia's Toolforge cluster did not come back after the maintain-kubeusers process restarted it at 18:30. To monitoring and to users the whole cluster looked down: Toolforge users could not launch tools with the webservice command on the Kubernetes backend or use kubectl, and an Icinga check against the "All k8s nodes healthy" endpoint started alerting at 18:45.

   Control plane unresponsiveService outageTLS certificate errors

   Trigger

   At 18:30 the maintain-kubeusers process restarted the Kubernetes API server, which forced it to reconnect to etcd with a CA certificate that Puppet changes the night before had left mismatched with the etcd servers.

   Root cause

   Configuration change

   Component

   Kubernetes API serveretcdConfiguration management tool

   Contributing factor

   Change with a delayed effectHarmful manual actionNo runbook

   Detection

   Automated monitoring: an Icinga alert on the toolschecker check http://checker.tools.wmflabs.org/k8s/nodes/ready, about "All k8s nodes healthy", fired at 18:45.

   Remediation

   • Rebooted the Kubernetes master and the etcd cluster, which did not help
   • Took a backup of etcd and reduced the etcd cluster to one node to rule out raft election flaps
   • Brought etcd back to full capacity through a full disaster recovery from the backup
   • Swapped the API server's CA certificate for the one from the etcd servers, which restored services until Puppet reverted it
   • Placed the etcd Puppet CA certificate in /var/lib/puppet/client/ssl/certs/ca.pem so Puppet kept the right CA

   Action items

   • Document the etcd cluster processes
   • Upgrade Toolforge Kubernetes to improve etcd and Kubernetes error reporting and stop needing an API server restart to add a tool
   • Make kube2proxy more resilient to API server failures
   • Audit the Puppet CA certificates so the tools project has no more than one
   • Make the Kubernetes control plane highly available

   Lessons

   Components that often misbehave, such as etcd, drew the responders to the wrong fix, and working on etcd instead of the CA certificates made the outage worse. Puppet-managed certificates are risky, and very old Kubernetes versions log errors that are hard to read. The failure was hard to find because it surfaced at an API server restart rather than at the Puppet change, because the etcd cluster had long-standing undocumented quirks, and because no disaster recovery or node replacement procedure was written down for etcd.

   View case

By root cause2