Incident Library

Component

Database failover manager

데이터베이스 장애 조치 관리자 · failover-manager

18 cases · 2011 to 2026

Root cause Software defect4Capacity exhaustion3Hardware failure3Configuration change2DNS or network fault2Cloud provider outage1Faulty deployment1Operator error1Breaking change in an upgrade1
Contributing factor Overlapping failures6Unverified assumption5Automation amplification5Harmful manual action3Monitoring blind spot3Unrepresentative staging3Excessive blast radius2Unreviewed defaults2Change with a delayed effect2No staged rollout2No headroom1Partial rollout1Single point of failure1Unsafe failure behaviour1Dependence on a warm cache1
Symptom Failover failure10High error rate6High latency6Service outage6Replication lag4Writes rejected4Intermittent timeouts3Split brain3Database overload2Storage disconnected or stale mounts2Data loss1Lock contention or deadlock1Monitoring outage1Network unreachable1Node disk full1Sustained high node CPU1Packet loss1Stale endpoint routing1Unnecessary failover1
Organisation GitHub2GitLab2Hatena2Cloudflare1DeNA1GoCardless1Hightouch1LINE1Microsoft Azure1Neon1Postmark1Pusher1Redis1Stripe1updown.io1

Cases, by severity

  1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026

    Root cause Breaking change in an upgrade

  2. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025

    Root cause Software defect

  3. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalledNeon · 2024

    Root cause Hardware failure

  4. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on themMicrosoft Azure · 2024

    Root cause Configuration change

  5. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutesGitLab · 2023

    Root cause Cloud provider outage

  6. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed overPusher · 2022

    Root cause Software defect

  7. IL-0321 GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021

    Root cause Capacity exhaustion

  8. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicasCloudflare · 2020

    Root cause DNS or network fault

  9. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019

    Root cause Software defect

  10. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went downupdown.io · 2019

    Root cause Hardware failure

  11. IL-0457 The HA HDFS NameNode of a log-collection Hadoop cluster failed over around 09:00 after its ZKFC health check timed outLINE · 2019

    Root cause Capacity exhaustion

  12. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hoursGitHub · 2018

    Root cause DNS or network fault

  13. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degradedHatena · 2018

    Root cause Configuration change

  14. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replicaGoCardless · 2017

    Root cause Hardware failure

  15. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017

    Root cause Operator error

  16. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failedRedis · 2016

    Root cause Faulty deployment

  17. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012

    Root cause Software defect

  18. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayedPostmark · 2011

    Root cause Capacity exhaustion

View on the graph