Cases2 cases

1 to 2 of 2

1. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
2. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours

   GitHub2018

   Symptoms

   At 22:54 UTC on 21 October 2018 GitHub's internal monitoring started alerting on many faults at once. By 23:02 UTC the first responders saw that the replication topologies of many MySQL clusters were in an unexpected state: the Orchestrator API showed topologies made only of servers in the US West Coast data center. When connectivity came back, the application tier had immediately started sending writes to the new West Coast primaries.

   Split brainFailover failureReplication lagHigh latency

   Trigger

   At 22:52 UTC on 21 October, routine maintenance to replace failing 100G optical equipment cut connectivity between GitHub's US East Coast network hub and its primary US East Coast data center for 43 seconds.

   Root cause

   DNS or network fault

   Component

   DatabaseDatabase failover managerDatabase replicationDatabase backups and snapshotsNIC, cable and switch port link

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   GitHub's internal monitoring raised alerts on numerous faults at 22:54 UTC, two minutes after the partition. By 23:02 UTC the first responders had found, by querying the Orchestrator API, that many database clusters had a topology with only US West Coast servers.

   Remediation

   • Locked the internal deployment tooling to stop further changes
   • Paused webhook delivery and GitHub Pages builds to stop jobs writing metadata
   • Restored the affected MySQL clusters from backups into the US East Coast data center and let them replicate from the West Coast
   • Found ways to restore directly from the West Coast to get around the throughput limits of downloading from off-site storage
   • Moved all database primaries back to the US East Coast data center
   • Spread read traffic over additional MySQL read replicas in the US East Coast public cloud so replication could catch up
   • Failed over to the original topology once the replicas were in sync
   • Processed the backlog of webhook events and Pages builds, pausing to raise an internal TTL after about 200,000 webhook payloads were dropped

   Action items

   • Reconcile the writes from the captured MySQL binary logs that were not replicated, automatically where possible and with user outreach otherwise
   • Make sure status updates can be published even when GitHub Pages builds are paused
   • Give more accurate public estimates of the time to repair
   • Configure Orchestrator so it cannot promote database primaries across regional boundaries
   • Move to a status reporting mechanism that shows the status of each component of the platform
   • Serve GitHub traffic from multiple data centers in an active/active/active design with N+1 redundancy at the facility level
   • Test assumptions about the system more proactively
   • Validate failure scenarios systematically, with investment in fault injection and chaos engineering tooling

   Lessons

   Orchestrator did what it was configured to do, but the application tier could not work with primaries on the other side of the country: leader election within one region is generally safe, while the sudden cross-country latency was a major contributing factor. The behaviour emerged from a partition larger than any GitHub had seen. Replication caught up along a power decay curve, not a straight line, so the linear estimates of recovery time were wrong. GitHub concluded that tighter operational controls and faster response are not enough on their own and that failure scenarios have to be validated before they happen.

   View case

By root cause1