Incident Library

IL-0303 GitHub · 2012-09-10database

GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down

In mid-August 2012 GitHub replaced its pair of DRBD-backed MySQL servers with a three-node cluster. Pacemaker and Heartbeat manage two virtual IPs, one read/write and one read-only, and Percona Replication Manager, a resource agent for Pacemaker, moves the 'active' role (the master that takes reads and writes) and the 'standby' role (a read-only slave) between the nodes. MySQL runs on every node, so a failover only moves a virtual IP after flushing transactions and changing the read_only variable.

On September 10th a schema migration loaded the master so much that the replication manager's health checks failed; it moved the active role to another node and stopped MySQL on the original one. The new master had a cold InnoDB buffer pool, failed its health checks under the site's load, and the role moved back. The team enabled Pacemaker's maintenance-mode to stop all health checks and automatic failover, and performance recovered as the buffer pool warmed. Maintenance-mode also kept the standby from switching to its new master and replicating, so it returned incorrect results. On September 11th, an attempt to disable maintenance-mode caused a Pacemaker segfault and a partition of the cluster state; two master elections ran without coordination, and the out-of-date node 'c' became master by itself at 8:19 AM. The team powered it off at 8:26 AM, which took github.com down, could not change the cluster state afterwards, and so stopped Pacemaker and Heartbeat on nodes 'a' and 'b', started them on 'a', and started MySQL there in the active role. The site recovered as the buffer pool warmed.

GitHub uses Redis to look up dashboard event stream entries and repository routes by ids that MySQL generates. For records created while node 'c' was master, those ids no longer matched, so some events appeared on the wrong users' dashboards and some new repositories were routed wrongly. 16 of those repositories were private and could be reached by people outside their collaborators and teams for seven minutes, from 8:19 AM to 8:26 AM PDT on September 11th. GitHub named the automated failover of its main production database as the root cause of both outages, which together caused one hour and 46 minutes of downtime and another hour of significantly degraded performance.

Severity
–
Scope
GitHub.com had one hour and 46 minutes of downtime and another hour of significantly degraded performance over two outages; some events appeared on wrong dashboards and 16 private repositories were reachable by outsiders for seven minutes
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On Monday, September 10th, 2012, during a schema migration, the health checks on GitHub's MySQL master failed and Percona Replication Manager moved the 'active' role to another node of the new three-node cluster. That node's InnoDB buffer pool was cold and it performed poorly, its health checks failed in turn, and the role failed back to the original server. GitHub.com went down.

The migration, run with GitHub's two-pass system for zero-downtime MySQL schema changes, put more load on the database than the operations team had seen in earlier migrations of this kind. When the health checks failed, the replication manager also stopped MySQL on the node it judged failed. On the new master, the site's query load against a cold cache made the health checks fail again.

To stop the failovers the team put Pacemaker into maintenance-mode, in which no health checks or automatic failovers run, and the site recovered slowly as the buffer pool warmed. The next morning a developer reported incorrect query results from the 'standby' node: maintenance-mode had also blocked the steps that should have pointed the standby at its new master and started replication.

When the team tried to disable maintenance-mode on Tuesday, September 11th, Pacemaker segfaulted and the cluster state split, with nodes 'a' and 'b' rejecting most messages from node 'c' and the other way round, although the cluster was configured to need a majority before acting. Node 'c', the out-of-date one, was elected master on its own at 8:19 AM. The team powered it off at 8:26 AM to stop further data drift, which cut all production database access and took down all of github.com.

Symptom Failover failureDatabase overloadHigh latencyReplication lagSplit brainService outage

Root cause

Root cause Software defect

Component DatabaseDatabase failover managerDatabase replication
Contributing factor Automation amplificationDependence on a warm cacheHarmful manual actionOverlapping failures
Trigger
A schema migration on Monday, September 10th put more load on the master database than earlier migrations had, and Percona Replication Manager's health checks on the master failed.

Detection

On the morning of September 11th, a developer told the operations team that the 'standby' node was returning incorrect query results.

Remediation

Action items

Lessons

GitHub concluded that automated failover is not the right strategy for its primary production database: for every failover in this incident, any member of the operations team would have said it should not happen. Failovers that must happen, in an emergency or during maintenance, should not hurt performance.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database failover manager, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.