Cases2 cases

1 to 2 of 2

1. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
2. IL-0296 Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customers

   Transloadit2016

   Symptoms

   Reports came in of more WORKER_JOB_ERROR errors, which tell Transloadit there is a communication problem between its Uploaders, the machines that receive files and orchestrate Assemblies, and its Drones, the machines that run encoding tasks. Many Assemblies became much slower because retries with exponential back-off kicked in, and a few could not be replayed.

   High error rateHigh latencyLoad imbalance

   Trigger

   A steep increase in HLS traffic, together with more Drones online processing the same volume faster, which put more load on the central Redis nodes.

   Root cause

   Capacity exhaustion

   Component

   In-memory cacheDatabase replication

   Contributing factor

   Monitoring blind spotUnverified assumptionHarmful manual action

   Detection

   Reports of an increased number of WORKER_JOB_ERROR errors.

   Remediation

   • Re-added lost messages and relaxed heartbeat monitoring as a stopgap
   • Upgraded Redis to the latest stable version
   • Doubled the Redis cluster to six nodes and then added three more, which made it worse
   • Brought the number of read replicas down to one, which cut traffic and failure rates
   • Capped how far Drones scale up in each region

   Action items

   • Add logging and alarms for approaching saturation
   • Find ways to decentralise the traffic between the Redis nodes and the Drones and Uploaders
   • Add sharding per Uploader on top of the sharding per region

   Lessons

   To handle more traffic Transloadit had to use fewer Redis nodes, not more: each added replica multiplied the replication traffic from the single leader that takes every write, so scaling up the number of nodes was the wrong call. Bandwidth had not been among the monitored Redis metrics.

   View case

By root cause2