Incident Library

Symptom

Failover failure

장애 조치 실패 · failover-failed

16 cases · 2011 to 2025

Root cause Software defect4Capacity exhaustion3Cloud provider outage2Hardware failure2Configuration change1Faulty deployment1DNS or network fault1Operator error1unknown1
Component Database13Database failover manager10In-memory cache4Cloud block storage2Database replication2Message broker2CDN and edge network1Coordination and key-value store1Database backups and snapshots1Database proxy and connection pooler1Authoritative DNS servers and records1NIC, cable and switch port link1
Contributing factor Excessive blast radius5Monitoring blind spot5Overlapping failures5Unverified assumption3Automation amplification3Harmful manual action3Single point of failure3Signal buried in noise1Change during peak hours1Change with a delayed effect1Inefficient query1Unpatched known bug1No headroom1No staged rollout1Retry storm1Unrepresentative staging1Unsafe failure behaviour1Dependence on a warm cache1
Organisation GitHub3Amazon Web Services1AppOptics1Discord1GitLab1GoCardless1Hightouch1Microsoft Azure1Postmark1Redis1Stack Exchange1Stripe1updown.io1Wasabi1

Cases

  1. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025

    Root cause Software defect

  2. IL-0544 Wasabi's database cache stopped responding when its queueing service filled up, and S3 failed in all regionsWasabi · 2024

    Root cause Capacity exhaustion

  3. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on themMicrosoft Azure · 2024

    Root cause Configuration change

  4. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutesGitLab · 2023

    Root cause Cloud provider outage

  5. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022GitHub · 2022

    Root cause Capacity exhaustion

  6. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019

    Root cause Software defect

  7. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went downupdown.io · 2019

    Root cause Hardware failure

  8. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hoursGitHub · 2018

    Root cause DNS or network fault

  9. IL-0588 AppOptics metrics APIs returned 5xx as Kafka partitions stayed leaderless after an AWS power event hit their ZooKeeperAppOptics · 2018

    Root cause Cloud provider outage

  10. IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017

    Root cause Software defect

  11. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replicaGoCardless · 2017

    Root cause Hardware failure

  12. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutesStack Exchange · 2017

    Root cause unknown

  13. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failedRedis · 2016

    Root cause Faulty deployment

  14. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012

    Root cause Software defect

  15. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011

    Root cause Capacity exhaustion

  16. IL-0337 EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011

    Root cause Operator error

View on the graph