IL-0240 Cloudflare · 2020-11-02databasenetworking
Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas
Cloudflare's control plane, the API and dashboard, is made of many microservices that are redundant across two regions, and most of their databases can be written in only one region at a time. Each rack in the control plane data centers has a pair of switches that are both normally active. On 2 November 2020 at 14:43 UTC one switch went into a partly working state in which some of its control protocols kept running, vPC did not, and its data plane dropped part of the traffic. Because the links stayed up, the servers did not fail over to the peer switch.
One server of the etcd cluster was in that rack. Traffic between node 1 and the leader, node 3, passed through the bad switch, while node 1 could still reach node 2 and node 2 could reach node 3. Node 1 kept starting elections and voting for itself while node 2 kept voting for node 3, so the elections tied, and because Raft blocks writes during an election, etcd was read-only until the switch recovered. Cloudflare's database cluster management system uses etcd for member discovery and coordination, so two clusters could not report a healthy primary, and the system automatically promoted their synchronous replicas, without error or data loss. A defect in the cluster management system required every replica to be rebuilt after a promotion. One cluster recovered quickly. The other handles authentication for API calls and the dashboard, takes heavy reads and relied on one replica to spare the primary; with no replica available, the primary took all the load and was overloaded, which is when the main impact began.
Engineers shed load from the authentication database by slowing the push of SSL certificates to the edge, email sending and other features, and moved API read traffic by hand to the replicas in the secondary data center, which are not part of local failover and stayed online. This improved the API but hurt the dashboard, whose sessions could not cross data centers, so authentication calls were sent fully back to the primary data center. Because of the database's size, rebuilding a replica took hours; at 21:20 UTC the first rebuilt replica put itself into service and performance returned to normal. The services that had been turned down were ramped back up so asynchronous work could catch up. In a postscript Cloudflare noted that the distributed systems community saw the switch failure as an omission fault rather than a Byzantine fault.
- Severity
- –
- Scope
- Cloudflare control plane: the API, whose query success rate dipped as low as 75%, and the dashboard, up to 80 times slower than normal; the edge kept working
- Users affected
- –
- Region
- –
- Time to detect
- 0m
- Time to resolve
- 6h 33m
Symptoms
On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.
The first signal came at 14:43 UTC, when alerts fired because a network switch in one rack of a control plane data center stopped answering pings. The switch had only partly failed: LACP and BGP kept running while vPC, which keeps the ports of a switch pair in step, did not, and its data plane forwarded only some of the packets it received. Because LACP spreads traffic over the links, each connected server saw trouble on only part of its traffic, and since the links never went down, traffic did not move to the peer switch. The switch recovered by itself six minutes later.
From 14:44 UTC the etcd cluster, which had one server in that rack, kept holding leader elections that never produced a leader every member could reach, so etcd stayed read-only until the switch recovered. At 14:45 UTC two database clusters could no longer report a healthy primary through etcd, and the cluster management system promoted a synchronous replica in each. In the cluster that handles authentication for API calls and dashboard activity, no replica was left online after the promotion, and the new primary was overwhelmed by the read load a replica had carried before.
Moving API reads to replicas in the secondary data center improved API availability but made the dashboard worse, because dashboard user sessions, kept in a Redis cluster, could not move between data centers without disruption. Performance returned to normal only at 21:20 UTC, when the first rebuilt replica came back into service.
Symptom High error rateHigh latencyDatabase overloadWrites rejectedPacket loss
Root cause
Root cause DNS or network fault
- Trigger
- At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.
Detection
At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.
Remediation
- Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
- Moved API traffic that could use read replicas to the secondary data center by hand
- Sent authentication calls fully back to the primary data center when the dashboard got worse
- Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
- Ramped the turned-down services back up so asynchronous processing could catch up
Action items
- Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
- Redesign the user sessions system so dashboard traffic can be steered between data centers
- Change the auto-remediation configuration so a database replica is not promoted so quickly
- Tune each system involved to be more resilient to failures of its dependencies
Lessons
Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.
Sources
Related cases
IL-0210GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hoursGitHub · 2018same root cause · shared Database, Database failover manager, Database replication, NIC, cable and switch port link
IL-0128Network interfaces that dropped for a few seconds made NCCL time out and stalled or slowed LLM training on a cluster of more than 10,000 GPUsByteDance · 2024same root cause · shared NIC, cable and switch port link
IL-0590F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittentlyMailgun · 2014same root cause · shared Database replication
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0240
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0240/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.