Cases6 cases

1 to 5 of 6

1. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over

   Pusher2022

   Symptoms

   At 10:42 UTC on 11 April 2022, the API for publishing messages to Pusher Channels on the mt1 cluster became largely unavailable, and from about 10:44 requests to the MT1 API returned more errors.

   Service outageHigh latencySplit brain

   Trigger

   An engineer restarted the master node of an mt1 Redis group to apply a larger replication buffer, after restarting both replicas of that group.

   Root cause

   Software defect

   Component

   In-memory cacheDatabase failover manager

   Contributing factor

   Unverified assumptionUnrepresentative staging

   Detection

   Pusher saw error rates rise for requests to the MT1 API from about 10:44 UTC.

   Action items

   • Fix the Redis shutdown script to check that failover has completed and the node is no longer a master before terminating

   View case
2. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
3. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down

   updown.io2019

   Symptoms

   At 2:30 am CEST on 3 April 2019, the primary MongoDB server of updown.io started answering queries about 200 times slower than usual, and the website and the monitoring daemons went almost completely down.

   High latencyService outageFailover failure

   Trigger

   One of the three SSDs in the primary MongoDB server's RAID5 array began to fail, slowing the server's IO without taking the server down.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   –

   Detection

   The founder, who was asleep when the slowdown began, noticed the problem at 8:45 am CEST and found the IO performance issue by inspecting the primary server.

   Remediation

   • Forced the primary to step down after making sure the secondary was ready to take over
   • Took the failing SSD out of the RAID array, which restored the old primary's performance
   • Asked the hosting provider to replace the SSD
   • Replaced the less powerful secondary with a more powerful server

   Action items

   • Write a script that detects when MongoDB is up but not responsive and forces a step-down automatically
   • Shut down and delete the old secondary once nothing is left to investigate on it
   • Make sure the new secondary can run as primary for several days
   • Test the automatic step-down script once the primary server is healthy again
   • Provide a proper status page for updown.io
   • Replace the custom script with MongoDB's Storage Node Watchdog

   View case
4. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours

   GitHub2018

   Symptoms

   At 22:54 UTC on 21 October 2018 GitHub's internal monitoring started alerting on many faults at once. By 23:02 UTC the first responders saw that the replication topologies of many MySQL clusters were in an unexpected state: the Orchestrator API showed topologies made only of servers in the US West Coast data center. When connectivity came back, the application tier had immediately started sending writes to the new West Coast primaries.

   Split brainFailover failureReplication lagHigh latency

   Trigger

   At 22:52 UTC on 21 October, routine maintenance to replace failing 100G optical equipment cut connectivity between GitHub's US East Coast network hub and its primary US East Coast data center for 43 seconds.

   Root cause

   DNS or network fault

   Component

   DatabaseDatabase failover managerDatabase replicationDatabase backups and snapshotsNIC, cable and switch port link

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   GitHub's internal monitoring raised alerts on numerous faults at 22:54 UTC, two minutes after the partition. By 23:02 UTC the first responders had found, by querying the Orchestrator API, that many database clusters had a topology with only US West Coast servers.

   Remediation

   • Locked the internal deployment tooling to stop further changes
   • Paused webhook delivery and GitHub Pages builds to stop jobs writing metadata
   • Restored the affected MySQL clusters from backups into the US East Coast data center and let them replicate from the West Coast
   • Found ways to restore directly from the West Coast to get around the throughput limits of downloading from off-site storage
   • Moved all database primaries back to the US East Coast data center
   • Spread read traffic over additional MySQL read replicas in the US East Coast public cloud so replication could catch up
   • Failed over to the original topology once the replicas were in sync
   • Processed the backlog of webhook events and Pages builds, pausing to raise an internal TTL after about 200,000 webhook payloads were dropped

   Action items

   • Reconcile the writes from the captured MySQL binary logs that were not replicated, automatically where possible and with user outreach otherwise
   • Make sure status updates can be published even when GitHub Pages builds are paused
   • Give more accurate public estimates of the time to repair
   • Configure Orchestrator so it cannot promote database primaries across regional boundaries
   • Move to a status reporting mechanism that shows the status of each component of the platform
   • Serve GitHub traffic from multiple data centers in an active/active/active design with N+1 redundancy at the facility level
   • Test assumptions about the system more proactively
   • Validate failure scenarios systematically, with investment in fault injection and chaos engineering tooling

   Lessons

   Orchestrator did what it was configured to do, but the application tier could not work with primaries on the other side of the country: leader election within one region is generally safe, while the sudden cross-country latency was a major contributing factor. The behaviour emerged from a partition larger than any GitHub had seen. Replication caught up along a power decay curve, not a straight line, so the linear estimates of recovery time were wrong. GitHub concluded that tighter operational controls and faster response are not enough on their own and that failure scenarios have to be validated before they happen.

   View case
5. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degraded

   Hatena2018

   Symptoms

   On 26 September 2018 at 10:51 JST the whole Mackerel monitoring service stopped working properly and its alive monitoring stopped. The application servers could not connect to Redis, which stores Mackerel's monitoring data, and could not return correct responses.

   Monitoring outageHigh latencyIntermittent timeouts

   Trigger

   Redis memory use had been growing, so operators set up replication in order to scale Redis up. Building the replication took a long time and Redis stopped answering for a while, and keepalived treated this as a node failure and started a failover nobody intended.

   Root cause

   Configuration change

   Component

   In-memory cacheDatabase replicationDatabase failover manager

   Contributing factor

   Automation amplificationOverlapping failures

   Remediation

   • Switched back to the correct Redis and briefly saw the application return
   • Restarted the application servers one after another, which did not improve the symptoms
   • Detected the organizations posting inappropriate metrics and blocked their requests
   • Adjusted the timeout interval of the application servers
   • Temporarily added capacity to Diamond, the TSDB
   • Increased the number of Kinesis shards and Redis Cluster shards
   • Scaled out the API application servers
   • Put Mackerel in maintenance mode, checked it internally and let outside requests back gradually

   Action items

   • Raise the number of keepalived health checks for Redis so a brief load increase does not trigger a failover
   • Scale Redis up
   • Add more application servers
   • Change the application to store only the monitoring data it needs in Redis, reducing Redis memory use
   • Build a way to detect inappropriate API requests quickly and show them on a dashboard
   • Build a function to block organizations making inappropriate requests quickly
   • Keep improving the accuracy of detecting inappropriate requests and consider adding features such as an API limit
   • Strengthen monitoring of the application servers' internal processing so problems are handled before they become outages

   Lessons

   Hatena did not find why recovery took so long, but expects that keeping failover from reacting to short load spikes and giving the application more performance headroom would prevent an outage this long. It also aims to keep problems contained to a small part of the service when a failure does happen.

   View case

By root cause4