Cases18 cases

15 to 18 of 18

1. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of data

   Hatena2017

   Symptoms

   On 7 August 2017, while checking the application after the maintenance that moved Mackerel from an on-premises data center to AWS, the team found that data updated during a specific time window was missing. Everything written to the PostgreSQL database between 14:30 and 15:30 JST was gone.

   Data lossReplication lagNetwork unreachable

   Trigger

   One hour after the application was switched to the newly promoted primary db001, db001 lost network connectivity and the database failed over to the standby db002.

   Root cause

   Operator error

   Component

   DatabaseDatabase replicationDatabase failover manager

   Contributing factor

   Overlapping failuresUnreviewed defaultsUnrepresentative stagingAutomation amplification

   Detection

   The team found the missing data while checking how the application behaved after the migration work.

   Remediation

   • Tried to restore the lost updates by hand from the table data left on db001 and gave up
   • Tried point-in-time recovery from the base backup and WAL and gave up
   • Built a new replica db003 with db002 as primary to restore redundancy
   • Kept db001 in maintenance state for investigation
   • Watched db002 for some time to confirm the problem did not recur, then completed the migration

   Action items

   • Change the PostgreSQL replica creation script to set recovery_target_timeline = 'latest' in recovery.conf by default
   • Add a check to a server configuration test tool such as Serverspec that recovery.conf contains recovery_target_timeline = 'latest'

   Lessons

   A setting needed only for a rare layout, such as a grandchild replica during a migration, was applied by hand through a script option and was forgotten. Because recovery_target_timeline = 'latest' is harmless on ordinary standbys too, Hatena now sets it by default whenever a replica is created. The staging rehearsal used the same layout and steps but had the setting right, so it could not catch the mistake. Hatena notes that a similar layout may be used again for a future PostgreSQL upgrade or a move to another region.

   View case
2. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed

   Redis2016

   Symptoms

   On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

   Service outageFailover failure

   Trigger

   The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

   Root cause

   Faulty deployment

   Component

   In-memory cacheDatabase failover managerAuthoritative DNS servers and records

   Contributing factor

   Overlapping failures

   Remediation

   • Shut down and terminated all nodes of the cluster
   • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
   • Provisioned new cluster nodes from scratch without the weekly update
   • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
   • Bootstrapped and launched the databases, recovering the ones with persistence from storage

   Action items

   • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
   • Revalidate the QA processes for DevOps code deployments
   • Improve the tooling so it handles throttling of DNS requests

   View case
3. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down

   GitHub2012

   Symptoms

   On Monday, September 10th, 2012, during a schema migration, the health checks on GitHub's MySQL master failed and Percona Replication Manager moved the 'active' role to another node of the new three-node cluster. That node's InnoDB buffer pool was cold and it performed poorly, its health checks failed in turn, and the role failed back to the original server. GitHub.com went down.

   Failover failureDatabase overloadHigh latencyReplication lagSplit brainService outage

   Trigger

   A schema migration on Monday, September 10th put more load on the master database than earlier migrations had, and Percona Replication Manager's health checks on the master failed.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover managerDatabase replication

   Contributing factor

   Automation amplificationDependence on a warm cacheHarmful manual actionOverlapping failures

   Detection

   On the morning of September 11th, a developer told the operations team that the 'standby' node was returning incorrect query results.

   Remediation

   • Enabled Pacemaker's maintenance-mode so no health checks or automatic failovers ran, and let the buffer pool warm
   • Powered off the out-of-date node 'c' to end the partition
   • Stopped Pacemaker and Heartbeat on nodes 'a' and 'b' and started them on node 'a' only
   • Started MySQL on node 'a' in the active role
   • Removed the events leaked to wrong dashboards and audited the wrongly routed repositories
   • Contacted the owners of the exposed private repositories

   Action items

   • Change the Pacemaker configuration so the active database role fails over only when an operations engineer starts it
   • Find a way to warm the InnoDB buffer pool of slave databases so failovers do not hurt performance
   • Audit the Pacemaker and Heartbeat stack, focusing on the code path behind the segfault
   • Run hardware tests on the server where the segfault happened

   Lessons

   GitHub concluded that automated failover is not the right strategy for its primary production database: for every failover in this incident, any member of the operations team would have said it should not happen. Failovers that must happen, in an emergency or during maintenance, should not hurt performance.

   View case
4. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed

   Postmark2011

   Symptoms

   At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

   Node disk fullWrites rejectedFailover failure

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

   Remediation

   • Tried to free up logs and disk space on the primary server
   • Failed over manually to the second MySQL server
   • Recovered the jobs that had failed
   • Sent the emails that each server had queued locally while the database was unavailable

   Action items

   • Upgrade the MySQL cluster with a focus on availability and performance
   • Make the alerts more sensitive to catch problems sooner

   Lessons

   Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

   View case

By root cause9