Incident Library
Applied filters ComponentDatabase failover managerClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases18 cases

18 to 18 of 18

  1. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed
    Postmark2011
    Symptoms

    At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

    Node disk fullWrites rejectedFailover failure
    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase failover manager
    Contributing factor
    –
    Detection

    Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

    Remediation
    • Tried to free up logs and disk space on the primary server
    • Failed over manually to the second MySQL server
    • Recovered the jobs that had failed
    • Sent the emails that each server had queued locally while the database was unavailable
    Action items
    • Upgrade the MySQL cluster with a focus on availability and performance
    • Make the alerts more sensitive to catch problems sooner
    Lessons

    Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

    View case
Check your cluster for this

These 18 cases involve Database failover manager and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause9

Breaking change in an upgrade1 case

Component
DatabaseDatabase replicationDatabase failover manager
Contributing factor
Change with a delayed effect
Organisation
DeNA
  1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch up
    DeNA2026

Software defect4 cases

Component
Database failover manager4Database3In-memory cache1Database replication1
Contributing factor
Unverified assumption2Harmful manual action2Overlapping failures2Unrepresentative staging2Automation amplification1Excessive blast radius1Change with a delayed effect1Monitoring blind spot1Dependence on a warm cache1
Organisation
GitHub1Hightouch1Pusher1Stripe1
  1. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions
    Hightouch2025
  2. IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed over
    Pusher2022
  3. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day
    Stripe2019
  4. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down
    GitHub2012

Hardware failure3 cases

Component
Database3Database failover manager3
Contributing factor
Unverified assumption1Overlapping failures1Partial rollout1
Organisation
GoCardless1Neon1updown.io1
  1. IL-0306 A Neon pageserver's EC2 instance failed without warning and Postgres databases of 0.4% of us-east-1 projects stalled
    Neon2024
  2. IL-0289 A failing SSD made updown.io's primary MongoDB about 200 times slower, no failover happened and the site went down
    updown.io2019
  3. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica
    GoCardless2017

Configuration change2 cases

Component
Database failover manager2Cloud block storage1In-memory cache1Database1Database replication1
Contributing factor
Automation amplification1Excessive blast radius1Monitoring blind spot1No staged rollout1Overlapping failures1Unsafe failure behaviour1
Organisation
Hatena1Microsoft Azure1
  1. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them
    Microsoft Azure2024
  2. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degraded
    Hatena2018

Cloud provider outage1 case

Component
In-memory cacheDatabase failover manager
Contributing factor
Monitoring blind spotSingle point of failure
Organisation
GitLab
  1. IL-0291 GitLab.com's redis-sidekiq Sentinel could not elect a primary after a GCP host error, so Sidekiq was down for 30 minutes
    GitLab2023

Capacity exhaustion3 cases

Component
Database failover manager3Database2Cloud block storage1Shared network filesystem1
Contributing factor
Unverified assumption1Unreviewed defaults1No staged rollout1
Organisation
GitLab1LINE1Postmark1
  1. IL-0321 GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spiked
    GitLab2021
  2. IL-0457 The HA HDFS NameNode of a log-collection Hadoop cluster failed over around 09:00 after its ZKFC health check timed out
    LINE2019
  3. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed
    Postmark2011

DNS or network fault2 cases

Component
Database2Database replication2Database failover manager2NIC, cable and switch port link2Coordination and key-value store1Database backups and snapshots1
Contributing factor
Automation amplification2Unverified assumption1Harmful manual action1No headroom1
Organisation
Cloudflare1GitHub1
  1. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas
    Cloudflare2020
  2. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours
    GitHub2018

Operator error1 case

Component
DatabaseDatabase replicationDatabase failover manager
Contributing factor
Automation amplificationUnreviewed defaultsOverlapping failuresUnrepresentative staging
Organisation
Hatena
  1. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of data
    Hatena2017

Faulty deployment1 case

Component
In-memory cacheAuthoritative DNS servers and recordsDatabase failover manager
Contributing factor
Overlapping failures
Organisation
Redis
  1. IL-0278 A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed
    Redis2016