Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases33 cases

33 to 33 of 33

  1. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down
    GitHub2012
    Symptoms

    On Monday, September 10th, 2012, during a schema migration, the health checks on GitHub's MySQL master failed and Percona Replication Manager moved the 'active' role to another node of the new three-node cluster. That node's InnoDB buffer pool was cold and it performed poorly, its health checks failed in turn, and the role failed back to the original server. GitHub.com went down.

    Failover failureDatabase overloadHigh latencyReplication lagSplit brainService outage
    Trigger

    A schema migration on Monday, September 10th put more load on the master database than earlier migrations had, and Percona Replication Manager's health checks on the master failed.

    Root cause
    Software defect
    Component
    DatabaseDatabase failover managerDatabase replication
    Contributing factor
    Automation amplificationDependence on a warm cacheHarmful manual actionOverlapping failures
    Detection

    On the morning of September 11th, a developer told the operations team that the 'standby' node was returning incorrect query results.

    Remediation
    • Enabled Pacemaker's maintenance-mode so no health checks or automatic failovers ran, and let the buffer pool warm
    • Powered off the out-of-date node 'c' to end the partition
    • Stopped Pacemaker and Heartbeat on nodes 'a' and 'b' and started them on node 'a' only
    • Started MySQL on node 'a' in the active role
    • Removed the events leaked to wrong dashboards and audited the wrongly routed repositories
    • Contacted the owners of the exposed private repositories
    Action items
    • Change the Pacemaker configuration so the active database role fails over only when an operations engineer starts it
    • Find a way to warm the InnoDB buffer pool of slave databases so failovers do not hurt performance
    • Audit the Pacemaker and Heartbeat stack, focusing on the code path behind the segfault
    • Run hardware tests on the server where the segfault happened
    Lessons

    GitHub concluded that automated failover is not the right strategy for its primary production database: for every failover in this incident, any member of the operations team would have said it should not happen. Failovers that must happen, in an emergency or during maintenance, should not hurt performance.

    View case
Check your cluster for this

These 33 cases involve Database replication and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause11

Capacity exhaustion6 cases

Component
Database replication6Database5In-memory cache1Coordination and key-value store1Database backups and snapshots1Liveness and readiness probes1Pod scheduling and placement1Telemetry agent1
Contributing factor
Monitoring blind spot5Unverified assumption1Automation amplification1Untested backup1Unreviewed defaults1Ignored early signal1Unknown limit1Limited debug access1Harmful manual action1No runbook1No headroom1
Organisation
CDN771InfluxData1Instapaper1Intercom1Repro1Transloadit1
  1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restarting
    InfluxData2026
  2. IL-0288 ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipeline
    CDN772026
  3. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data
    Intercom2026
  4. IL-0443 A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or more
    Repro2019
  5. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017
  6. IL-0296 Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customers
    Transloadit2016

Breaking change in an upgrade2 cases

Component
Database2Database replication2PostgreSQL vacuum1Database proxy and connection pooler1Database failover manager1Query planner and table statistics1
Contributing factor
Unverified assumption1Automation amplification1Change with a delayed effect1Monitoring blind spot1Unrepresentative staging1
Organisation
DeNA1RevenueCat1
  1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch up
    DeNA2026
  2. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow
    RevenueCat2022

Certificate or credential expiry1 case

Component
DatabaseDatabase replication
Contributing factor
Unverified assumptionChange with a delayed effectMonitoring blind spotNo safeguard on destructive change
Organisation
LootLocker
  1. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours
    LootLocker2026

Software defect5 cases

Component
Database replication5Database4In-memory cache1Database backups and snapshots1Database failover manager1Object storage1
Contributing factor
Automation amplification2Untested backup1Harmful manual action1Monitoring blind spot1Overlapping failures1Pages not delivered1Single point of failure1Dependence on a warm cache1
Organisation
GitHub1Kickstarter1OpenStreetMap Foundation1PostHog1Upstash1
  1. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 days
    PostHog2026
  2. IL-0527 OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc error
    OpenStreetMap Foundation2025
  3. IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databases
    Upstash2025
  4. IL-0589 Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings table
    Kickstarter2013
  5. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down
    GitHub2012

Cloud provider outage2 cases

Component
Database2Database replication2Cloud block storage1Database proxy and connection pooler1Object storage1
Contributing factor
Ignored early signal1
Organisation
Kit1PlanetScale1
  1. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage
    PlanetScale2025
  2. IL-0488 Kit's RDS MySQL write latency rose 100x and running threads jumped from 9 to 900, backing up Sidekiq job queues
    Kit2022

Operator error4 cases

Component
Database4Database replication4Database backups and snapshots3Cloud block storage1Database failover manager1
Contributing factor
Unverified assumption2Untested backup2Unreviewed defaults2Overlapping failures2Automation amplification1Unpatched known bug1Harmful manual action1No runbook1No safeguard on destructive change1Pages not delivered1Single point of failure1Unrepresentative staging1
Organisation
GitLab1Hatena1Matrix.org Foundation1Oregon State University Open Source Lab1
  1. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours
    Matrix.org Foundation2025
  2. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of data
    Hatena2017
  3. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours
    GitLab2017
  4. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-master
    Oregon State University Open Source Lab2015

Dependency failure2 cases

Component
Database replication2Backbone network (WAN)1BGP routing1In-memory cache1ISP and transit networks1
Contributing factor
Single point of failure2Signal buried in noise1No runbook1Retry storm1
Organisation
37signals1OpenStreetMap Foundation1
  1. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks
    OpenStreetMap Foundation2024
  2. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020

Configuration change6 cases

Component
Database replication6Database3In-memory cache2Database backups and snapshots2Coordination and key-value store1Database failover manager1Service discovery1
Contributing factor
Harmful manual action2Automation amplification1Excessive blast radius1Monitoring blind spot1No safeguard on destructive change1Overlapping failures1Retry storm1Gap in automated tests1Unsafe failure behaviour1
Organisation
DigitalOcean2Hatena1Razorpay1Twilio1Wikimedia Foundation1
  1. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook run
    Wikimedia Foundation2024
  2. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024
  3. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicas
    Razorpay2020
  4. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degraded
    Hatena2018
  5. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017
  6. IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedly
    Twilio2013

Data loss or corruption1 case

Component
PostgreSQL vacuumDatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Monitoring blind spotNo headroomOverlapping failures
Organisation
BattleMetrics
  1. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates
    BattleMetrics2022

DNS or network fault3 cases

Component
Database replication3Database2Database failover manager2NIC, cable and switch port link2Cloud load balancer1Coordination and key-value store1Database backups and snapshots1
Contributing factor
Automation amplification2Unverified assumption1Harmful manual action1No headroom1
Organisation
Cloudflare1GitHub1Mailgun1
  1. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas
    Cloudflare2020
  2. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hours
    GitHub2018
  3. IL-0590 F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittently
    Mailgun2014

Hardware failure1 case

Component
DatabaseDatabase replication
Contributing factor
Harmful manual actionPages not delivered
Organisation
Wikimedia Foundation
  1. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only
    Wikimedia Foundation2019