IL-0210 GitHub · 2018-10-21databasenetworking

GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours

GitHub stores non-Git metadata, such as pull requests, issues, authentication and background job coordination, in several MySQL clusters split by function, from hundreds of gigabytes to nearly five terabytes, each with up to dozens of read replicas. Applications write to each cluster's primary and read mostly from replicas. Orchestrator, which is built on Raft consensus, manages the MySQL topologies and performs automated failover. During the 43-second partition between the US East Coast hub and the primary East Coast data center, the Orchestrator nodes in the US West Coast data center and in the US East Coast public cloud formed a quorum without the primary data center and failed clusters over so that writes went to the West Coast. When connectivity came back, the application tier immediately sent writes to the new West Coast primaries. A short window of East Coast writes had not been replicated, so both sides now held writes the other did not have.

Responders locked the internal deployment tooling, raised the site status to yellow and then red, and paged database engineers. Because the West Coast had taken more than 30 minutes of user writes, GitHub chose to fail forward to protect that data, accepting that the service would be unusable for many users. Webhook delivery and GitHub Pages builds were paused to stop jobs writing more metadata. From 00:41 UTC on 22 October the affected clusters were restored from backups, taken every four hours and kept in public cloud blob storage, into the East Coast data center, where they then replicated from the West Coast; later the teams found ways to restore directly from the West Coast to get around the off-site download limits. By 11:12 UTC all primaries were back in the East Coast, but many read replicas were hours behind, and with the morning load the lag grew. Read replicas that had been provisioned in the US East Coast public cloud spread the read load enough for replication to catch up.

At 16:24 UTC the replicas were in sync and GitHub failed over to the original topology, keeping the status red while it worked through more than five million queued hook events and 80 thousand Pages builds. About 200,000 webhook payloads had outlived an internal TTL and were dropped, so processing was paused and the TTL raised. At 23:03 UTC all pending webhooks and Pages builds were processed and the status went green, after 24 hours and 11 minutes of degradation. No user data was lost; GitHub captured the MySQL binary logs with the East Coast writes that had not reached the West Coast, for example 954 writes on one of the busiest clusters, and was reconciling them.

Severity: –
Scope: GitHub.com degraded for 24 hours and 11 minutes: out-of-date and inconsistent data, slow pages, and no webhook delivery or GitHub Pages builds for most of the incident; a few seconds of database writes needed manual reconciliation
Users affected: –
Region: US East Coast and US West Coast data centers
Time to detect: 2m
Time to resolve: 24h 11m

Symptoms

At 22:54 UTC on 21 October 2018 GitHub's internal monitoring started alerting on many faults at once. By 23:02 UTC the first responders saw that the replication topologies of many MySQL clusters were in an unexpected state: the Orchestrator API showed topologies made only of servers in the US West Coast data center. When connectivity came back, the application tier had immediately started sending writes to the new West Coast primaries.

For users, GitHub showed information that was out of date and inconsistent, and for most of the incident it did not deliver webhook events or build and publish GitHub Pages sites. Pages that had to write to a database over the cross-country link loaded slowly, because the applications in the US East Coast could not cope with the extra round-trip latency on most of their database calls.

Moving the primaries back to the US East Coast was not safe. The West Coast clusters had taken application writes for nearly 40 minutes, and the East Coast servers held several seconds of writes that had never reached the West Coast, so each side had writes the other lacked. Restoring the clusters from backups took hours, mostly in downloading multiple terabytes from a remote blob storage service and in decompressing, checksumming, preparing and loading the files onto new MySQL servers.

After all primaries were back in the East Coast at 11:12 UTC on 22 October, dozens of read replicas were still several hours behind the primary, and a request had a good chance of reading from one of them. The status page promised recovery in about two hours, from a linear estimate of replication progress, but the lag shrank along a power decay curve instead. As users in Europe and the US started their workday, write load grew and by 13:15 UTC the replication delay was growing rather than shrinking.

When processing of the queued work was re-enabled, about 200,000 webhook payloads had outlived an internal TTL and were dropped.

Symptom Split brain Failover failure Replication lag High latency

Root cause

Root cause DNS or network fault

Component Database Database failover manager Database replication Database backups and snapshots NIC, cable and switch port link

Contributing factor Automation amplification Unverified assumption

Trigger: At 22:52 UTC on 21 October, routine maintenance to replace failing 100G optical equipment cut connectivity between GitHub's US East Coast network hub and its primary US East Coast data center for 43 seconds.

Detection

GitHub's internal monitoring raised alerts on numerous faults at 22:54 UTC, two minutes after the partition. By 23:02 UTC the first responders had found, by querying the Orchestrator API, that many database clusters had a topology with only US West Coast servers.

Remediation

Locked the internal deployment tooling to stop further changes
Paused webhook delivery and GitHub Pages builds to stop jobs writing metadata
Restored the affected MySQL clusters from backups into the US East Coast data center and let them replicate from the West Coast
Found ways to restore directly from the West Coast to get around the throughput limits of downloading from off-site storage
Moved all database primaries back to the US East Coast data center
Spread read traffic over additional MySQL read replicas in the US East Coast public cloud so replication could catch up
Failed over to the original topology once the replicas were in sync
Processed the backlog of webhook events and Pages builds, pausing to raise an internal TTL after about 200,000 webhook payloads were dropped

Action items

Reconcile the writes from the captured MySQL binary logs that were not replicated, automatically where possible and with user outreach otherwise
Make sure status updates can be published even when GitHub Pages builds are paused
Give more accurate public estimates of the time to repair
Configure Orchestrator so it cannot promote database primaries across regional boundaries
Move to a status reporting mechanism that shows the status of each component of the platform
Serve GitHub traffic from multiple data centers in an active/active/active design with N+1 redundancy at the facility level
Test assumptions about the system more proactively
Validate failure scenarios systematically, with investment in fault injection and chaos engineering tooling

Lessons

Orchestrator did what it was configured to do, but the application tier could not work with primaries on the other side of the country: leader election within one region is generally safe, while the sudden cross-country latency was a major contributing factor. The behaviour emerged from a partition larger than any GitHub had seen. Replication caught up along a power decay curve, not a straight line, so the linear estimates of recovery time were wrong. GitHub concluded that tighter operational controls and faster response are not enough on their own and that failure scenarios have to be validated before they happen.

Sources

https://github.blog/news-insights/company-news/oct21-post-incident-analysis/ · blog · archived copy

Related cases

IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicasCloudflare · 2020
same root cause · shared Database failover manager, Database replication, Database, NIC, cable and switch port link
IL-0128 Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUsByteDance · 2024
same root cause · shared NIC, cable and switch port link
IL-0590 F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittentlyMailgun · 2014
same root cause · shared Database replication

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database failover manager, Database replication among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0210
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0210/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.