Cases10 cases

6 to 10 of 10

1. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours

   GitHub2018

   Symptoms

   At 22:54 UTC on 21 October 2018 GitHub's internal monitoring started alerting on many faults at once. By 23:02 UTC the first responders saw that the replication topologies of many MySQL clusters were in an unexpected state: the Orchestrator API showed topologies made only of servers in the US West Coast data center. When connectivity came back, the application tier had immediately started sending writes to the new West Coast primaries.

   Split brainFailover failureReplication lagHigh latency

   Trigger

   At 22:52 UTC on 21 October, routine maintenance to replace failing 100G optical equipment cut connectivity between GitHub's US East Coast network hub and its primary US East Coast data center for 43 seconds.

   Root cause

   DNS or network fault

   Component

   DatabaseDatabase failover managerDatabase replicationDatabase backups and snapshotsNIC, cable and switch port link

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   GitHub's internal monitoring raised alerts on numerous faults at 22:54 UTC, two minutes after the partition. By 23:02 UTC the first responders had found, by querying the Orchestrator API, that many database clusters had a topology with only US West Coast servers.

   Remediation

   • Locked the internal deployment tooling to stop further changes
   • Paused webhook delivery and GitHub Pages builds to stop jobs writing metadata
   • Restored the affected MySQL clusters from backups into the US East Coast data center and let them replicate from the West Coast
   • Found ways to restore directly from the West Coast to get around the throughput limits of downloading from off-site storage
   • Moved all database primaries back to the US East Coast data center
   • Spread read traffic over additional MySQL read replicas in the US East Coast public cloud so replication could catch up
   • Failed over to the original topology once the replicas were in sync
   • Processed the backlog of webhook events and Pages builds, pausing to raise an internal TTL after about 200,000 webhook payloads were dropped

   Action items

   • Reconcile the writes from the captured MySQL binary logs that were not replicated, automatically where possible and with user outreach otherwise
   • Make sure status updates can be published even when GitHub Pages builds are paused
   • Give more accurate public estimates of the time to repair
   • Configure Orchestrator so it cannot promote database primaries across regional boundaries
   • Move to a status reporting mechanism that shows the status of each component of the platform
   • Serve GitHub traffic from multiple data centers in an active/active/active design with N+1 redundancy at the facility level
   • Test assumptions about the system more proactively
   • Validate failure scenarios systematically, with investment in fault injection and chaos engineering tooling

   Lessons

   Orchestrator did what it was configured to do, but the application tier could not work with primaries on the other side of the country: leader election within one region is generally safe, while the sudden cross-country latency was a major contributing factor. The behaviour emerged from a partition larger than any GitHub had seen. Replication caught up along a power decay curve, not a straight line, so the linear estimates of recovery time were wrong. GitHub concluded that tighter operational controls and faster response are not enough on their own and that failure scenarios have to be validated before they happen.

   View case
2. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica

   GoCardless2017

   Symptoms

   At 15:09 BST on 10 October 2017, GoCardless's monitoring detected a total outage of its API and Dashboard: every request failed and returned an error. By 15:11 engineers saw evidence of a disk array failure on the primary Postgres node, but the Pacemaker cluster had not failed over to the synchronous replica as it should have.

   Service outageFailover failure

   Trigger

   The RAID controller of the primary Postgres node lost 3 disks of its array at once, so all reads and writes on the primary failed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Overlapping failuresUnverified assumption

   Detection

   GoCardless's monitoring detected the total outage of the API and Dashboard at 15:09 BST, and engineers began investigating.

   Remediation

   • Powered off the broken primary Postgres node
   • Cleared the Pacemaker error counts with crm resource cleanup, which had no effect
   • Edited recovery.conf on the synchronous replica under maintenance mode several times, which Pacemaker reverted
   • Put the cluster into maintenance mode, configured the synchronous replica as primary and started Postgres by hand
   • Reconfigured the backend applications to connect to the new primary's actual IP address
   • Bootstrapped a third node as an asynchronous replica
   • Added a VIP managed by hand by the infrastructure team for manual failover
   • Verified data integrity and found no corruption
   • Migrated to a new Pacemaker-managed database cluster in a maintenance window and decommissioned the old one

   Action items

   • Express the Backup VIP rule as an INF preference for a replica instead of a -INF preference against the primary
   • Replace the VIPs with proxies on the application servers that route traffic by the state of the cluster
   • Run harsher fault injection tests of the Postgres cluster and consider automation that injects failures continually
   • Consider adding arbitrary restrictions to game day exercises, such as automation that has failed and a problem that cannot be diagnosed, so engineers practise bringing the service back another way

   Lessons

   A rule that two resources must not run together does not behave like the opposite of a rule that they must run together, and in some failure conditions the difference stopped Pacemaker from promoting a replica. A misconfiguration that shows only when two processes crash at almost the same time will not appear in basic tests or daily operations, only through fault injection. After two years in which the automation had handled every failure, the team's skill at controlling the cluster by hand had faded, and a runbook unused for years would likely have been out of date.

   View case
3. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed

   Redis2016

   Symptoms

   On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

   Service outageFailover failure

   Trigger

   The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

   Root cause

   Faulty deployment

   Component

   In-memory cacheDatabase failover managerAuthoritative DNS servers and records

   Contributing factor

   Overlapping failures

   Remediation

   • Shut down and terminated all nodes of the cluster
   • Deleted all of the cluster's endpoints from its DNS zone, removing by hand the entries left by throttled deletes
   • Provisioned new cluster nodes from scratch without the weekly update
   • Reconnected the original nodes' storage to the new nodes so persistence-enabled databases could be recovered
   • Bootstrapped and launched the databases, recovering the ones with persistence from storage

   Action items

   • Suspend all automatic weekly upgrades until a procedure that prevents this kind of outage is in place
   • Revalidate the QA processes for DevOps code deployments
   • Improve the tooling so it handles throttling of DNS requests

   View case
4. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down

   GitHub2012

   Symptoms

   On Monday, September 10th, 2012, during a schema migration, the health checks on GitHub's MySQL master failed and Percona Replication Manager moved the 'active' role to another node of the new three-node cluster. That node's InnoDB buffer pool was cold and it performed poorly, its health checks failed in turn, and the role failed back to the original server. GitHub.com went down.

   Failover failureDatabase overloadHigh latencyReplication lagSplit brainService outage

   Trigger

   A schema migration on Monday, September 10th put more load on the master database than earlier migrations had, and Percona Replication Manager's health checks on the master failed.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover managerDatabase replication

   Contributing factor

   Automation amplificationDependence on a warm cacheHarmful manual actionOverlapping failures

   Detection

   On the morning of September 11th, a developer told the operations team that the 'standby' node was returning incorrect query results.

   Remediation

   • Enabled Pacemaker's maintenance-mode so no health checks or automatic failovers ran, and let the buffer pool warm
   • Powered off the out-of-date node 'c' to end the partition
   • Stopped Pacemaker and Heartbeat on nodes 'a' and 'b' and started them on node 'a' only
   • Started MySQL on node 'a' in the active role
   • Removed the events leaked to wrong dashboards and audited the wrongly routed repositories
   • Contacted the owners of the exposed private repositories

   Action items

   • Change the Pacemaker configuration so the active database role fails over only when an operations engineer starts it
   • Find a way to warm the InnoDB buffer pool of slave databases so failovers do not hurt performance
   • Audit the Pacemaker and Heartbeat stack, focusing on the code path behind the segfault
   • Run hardware tests on the server where the segfault happened

   Lessons

   GitHub concluded that automated failover is not the right strategy for its primary production database: for every failover in this incident, any member of the operations team would have said it should not happen. Failovers that must happen, in an emergency or during maintenance, should not hurt performance.

   View case
5. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed

   Postmark2011

   Symptoms

   At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

   Node disk fullWrites rejectedFailover failure

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

   Remediation

   • Tried to free up logs and disk space on the primary server
   • Failed over manually to the second MySQL server
   • Recovered the jobs that had failed
   • Sent the emails that each server had queued locally while the database was unavailable

   Action items

   • Upgrade the MySQL cluster with a focus on availability and performance
   • Make the alerts more sensitive to catch problems sooner

   Lessons

   Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

   View case

By root cause7