Cases6 cases

1 to 5 of 6

1. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook run

   Wikimedia Foundation2024

   Symptoms

   On 18 September 2024, while a cookbook was reconfiguring Wikimedia's core MariaDB databases from one-way eqiad -> codfw replication to circular replication ahead of a datacenter switchover, replication in the codfw -> eqiad direction broke on the x1 and es6 sections. Engineers watching the run noticed at 21:28 UTC that the x1 primary was not replicating.

   Replication lag

   Trigger

   Running the cookbook sre.switchdc.databases.prepare, which set up codfw -> eqiad replication and started pt-heartbeat, on the row-based x1 and es sections.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replication

   Contributing factor

   Monitoring blind spotGap in automated testsHarmful manual action

   Detection

   Engineers watching the first run of the automation saw on orchestrator that the x1 primary was not replicating, because monitoring of circular replication was not yet enabled. The break in the eqiad -> codfw direction was reported by Icinga.

   Remediation

   • Skipped replication of the heartbeat tables with Replication_wild_ignore_table='heartbeat.%'
   • Inserted the missing heartbeat row with binary logging off on the es hosts
   • Inserted the missing heartbeat row with binary logging off on the x1 codfw hosts and removed the ignore rule

   Action items

   • Avoid the same issue again, probably by inserting the missing row when it is detected
   • Add a post-maintenance health check that replication works in both directions
   • Add options to run the maintenance in another order or on each section individually
   • Update the Phabricator task for each section rather than only at the end
   • Fix the monitoring gaps for circular replication
   • Made pt-heartbeat-wikimedia use STATEMENT format for its session on the x1 and row-based read-write es hosts, fixed in a later deploy

   Lessons

   Circular replication is not a normal working mode, so the first fix only moved the break to the other side; TRUNCATE would have worked better than DELETE because it always replicates as a statement, but it was not considered because it is more dangerous. Running sql_skip_slave_counter was avoided, since it could have made things worse or lost data. The check of the replication channel should also run once all tasks are complete, not only right after setup. The combination of row-based replication, pt-heartbeat and differing heartbeat rows was not described in earlier maintenance or documentation.

   View case
2. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum

   DigitalOcean2024

   Symptoms

   From 16:30 UTC on 5 August 2024, DigitalOcean's internal services could not discover the other services they depended on. Creates, event processing and the management of DigitalOcean products failed in all regions through the Cloud Control Panel and the API, while existing customer resources such as Droplets stayed online.

   Service outageCluster or node provisioning stall

   Trigger

   During the rollout of a new datastore cluster in a new datacenter, a replication configuration was deployed against the datastore behind internal service discovery that gave the new datacenter, which held zero keys, 100% ownership of all keys.

   Root cause

   Configuration change

   Component

   Coordination and key-value storeService discoveryDatabase replication

   Contributing factor

   Excessive blast radius

   Detection

   Service discovery unavailability was first reported internally at 16:35 UTC, five minutes after the rollout of the new datastore cluster began.

   Remediation

   • Reverted the replication configuration and ran it against the region that had become the datastore owner
   • Applied the configuration by hand on the nodes of lagging regions and ran local data repairs
   • Cleaned up failed events and processed pending events that had not timed out

   Action items

   • Perform this kind of datastore extension inside a declared maintenance window, announced on the status page
   • Update the process documentation for this type of deployment, stage by stage
   • Automate the manual steps used in this deployment to reduce human error
   • Evaluate the internal datastore topology and whether to regionalise or layer it

   View case
3. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicas

   Razorpay2020

   Symptoms

   On a Friday evening in December 2019, Razorpay's internal monitoring alerted on an unusually high number of 5xx errors. The SRE team found that the main application, called API, could not connect to its RDS MySQL database, but within about 110 seconds the application recovered by itself and the alerts stopped. The RDS logs showed that the master instance had gone through a Multi-AZ failover.

   High error rateReplication lagData loss

   Trigger

   The master instance of the API application's RDS MySQL database went through a Multi-AZ failover to its standby in another availability zone.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replicationDatabase backups and snapshots

   Contributing factor

   –

   Detection

   An internal monitoring alert on unusually high 5xx errors, then alerts that replication had failed on all replicas. The data loss was found the next day by matching the database against the application trace logs.

   Remediation

   • Moved all read replica traffic to the master instance
   • Recreated all five read replicas after a six-hour full snapshot and moved traffic back to them
   • Recovered the missing queries from the binary logs and corrected the data stored in RDS
   • Changed innodb_flush_log_at_trx_commit on the master from 2 to 1

   Lessons

   With innodb_flush_log_at_trx_commit set to 2, Razorpay had been open to data loss on any failover all along, even though RDS Multi-AZ copies everything on the primary's disk to the standby. The incident also taught the team how RDS runs a Multi-AZ setup and its failovers.

   View case
4. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degraded

   Hatena2018

   Symptoms

   On 26 September 2018 at 10:51 JST the whole Mackerel monitoring service stopped working properly and its alive monitoring stopped. The application servers could not connect to Redis, which stores Mackerel's monitoring data, and could not return correct responses.

   Monitoring outageHigh latencyIntermittent timeouts

   Trigger

   Redis memory use had been growing, so operators set up replication in order to scale Redis up. Building the replication took a long time and Redis stopped answering for a while, and keepalived treated this as a node failure and started a failover nobody intended.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase replicationDatabase failover manager

   Contributing factor

   Automation amplificationOverlapping failures

   Remediation

   • Switched back to the correct Redis and briefly saw the application return
   • Restarted the application servers one after another, which did not improve the symptoms
   • Detected the organizations posting inappropriate metrics and blocked their requests
   • Adjusted the timeout interval of the application servers
   • Temporarily added capacity to Diamond, the TSDB
   • Increased the number of Kinesis shards and Redis Cluster shards
   • Scaled out the API application servers
   • Put Mackerel in maintenance mode, checked it internally and let outside requests back gradually

   Action items

   • Raise the number of keepalived health checks for Redis so a brief load increase does not trigger a failover
   • Scale Redis up
   • Add more application servers
   • Change the application to store only the monitoring data it needs in Redis, reducing Redis memory use
   • Build a way to detect inappropriate API requests quickly and show them on a dashboard
   • Build a function to block organizations making inappropriate requests quickly
   • Keep improving the accuracy of detecting inappropriate requests and consider adding features such as an API limit
   • Strengthen monitoring of the application servers' internal processing so problems are handled before they become outages

   Lessons

   Hatena did not find why recovery took so long, but expects that keeping failover from reacting to short load spikes and giving the application more performance headroom would prevent an outage this long. It also aims to keep problems contained to a small part of the service when a failure does happen.

   View case
5. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted

   DigitalOcean2017

   Symptoms

   At 10:24 AM EDT on April 5th, 2017, DigitalOcean began to receive alerts that its public services were not working, and by 10:27 the team had verified that the production database on the primary had been deleted. The control panel and the API were unavailable for four hours and fifty-six minutes: running Droplets kept working, but no new Droplets or other resources could be created or managed.

   Service outage

   Trigger

   A process performing automated testing had been misconfigured with production credentials, and the production primary database was deleted.

   Root cause

   Configuration change

   Component

   DatabaseDatabase replicationDatabase backups and snapshots

   Contributing factor

   No safeguard on destructive change

   Detection

   Alerts at 10:24 AM EDT that public services were not functioning; within three minutes the team found that the primary database had been deleted.

   Remediation

   • Began recovery from a time-delayed database replica
   • Took a backup of the time-delayed replica
   • Copied the backup to the primary and recovered it
   • Copied the backups to the replicas until all systems were restored

   Action items

   • Drastically reduce access to the primary system for certain actions
   • Upgrade network connectivity between database servers
   • Update hardware to speed up recovery
   • Assess further safeguards against developer error
   • Improve the processes around data recovery
   • Explore ways to provide better real-time information to customers during future customer-impacting events

   View case

By root cause1