Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases11 cases

11 to 11 of 11

  1. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down
    GitHub2012
    Symptoms

    On Monday, September 10th, 2012, during a schema migration, the health checks on GitHub's MySQL master failed and Percona Replication Manager moved the 'active' role to another node of the new three-node cluster. That node's InnoDB buffer pool was cold and it performed poorly, its health checks failed in turn, and the role failed back to the original server. GitHub.com went down.

    Failover failureDatabase overloadHigh latencyReplication lagSplit brainService outage
    Trigger

    A schema migration on Monday, September 10th put more load on the master database than earlier migrations had, and Percona Replication Manager's health checks on the master failed.

    Root cause
    Software defect
    Component
    DatabaseDatabase failover managerDatabase replication
    Contributing factor
    Automation amplificationDependence on a warm cacheHarmful manual actionOverlapping failures
    Detection

    On the morning of September 11th, a developer told the operations team that the 'standby' node was returning incorrect query results.

    Remediation
    • Enabled Pacemaker's maintenance-mode so no health checks or automatic failovers ran, and let the buffer pool warm
    • Powered off the out-of-date node 'c' to end the partition
    • Stopped Pacemaker and Heartbeat on nodes 'a' and 'b' and started them on node 'a' only
    • Started MySQL on node 'a' in the active role
    • Removed the events leaked to wrong dashboards and audited the wrongly routed repositories
    • Contacted the owners of the exposed private repositories
    Action items
    • Change the Pacemaker configuration so the active database role fails over only when an operations engineer starts it
    • Find a way to warm the InnoDB buffer pool of slave databases so failovers do not hurt performance
    • Audit the Pacemaker and Heartbeat stack, focusing on the code path behind the segfault
    • Run hardware tests on the server where the segfault happened
    Lessons

    GitHub concluded that automated failover is not the right strategy for its primary production database: for every failover in this incident, any member of the operations team would have said it should not happen. Failovers that must happen, in an emergency or during maintenance, should not hurt performance.

    View case
Check your cluster for this

These 11 cases involve Database replication and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Cloud provider outage1 case

Component
DatabaseDatabase proxy and connection poolerDatabase replicationObject storage
Contributing factor
–
Organisation
PlanetScale
  1. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage
    PlanetScale2025

Operator error3 cases

Component
Database3Database backups and snapshots3Database replication3Cloud block storage1
Contributing factor
Unverified assumption2Untested backup2Unreviewed defaults1Unpatched known bug1Harmful manual action1No runbook1No safeguard on destructive change1Overlapping failures1Pages not delivered1Single point of failure1
Organisation
GitLab1Matrix.org Foundation1Oregon State University Open Source Lab1
  1. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hours
    Matrix.org Foundation2025
  2. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours
    GitLab2017
  3. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-master
    Oregon State University Open Source Lab2015

Dependency failure1 case

Component
Database replicationISP and transit networks
Contributing factor
No runbookSingle point of failure
Organisation
OpenStreetMap Foundation
  1. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinks
    OpenStreetMap Foundation2024

Configuration change2 cases

Component
Database replication2Coordination and key-value store1Database1Database backups and snapshots1Service discovery1
Contributing factor
Excessive blast radius1No safeguard on destructive change1
Organisation
DigitalOcean2
  1. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024
  2. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deleted
    DigitalOcean2017

Breaking change in an upgrade1 case

Component
PostgreSQL vacuumDatabaseDatabase proxy and connection poolerDatabase replicationQuery planner and table statistics
Contributing factor
Unverified assumptionAutomation amplificationMonitoring blind spotUnrepresentative staging
Organisation
RevenueCat
  1. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow
    RevenueCat2022

Data loss or corruption1 case

Component
PostgreSQL vacuumDatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Monitoring blind spotNo headroomOverlapping failures
Organisation
BattleMetrics
  1. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates
    BattleMetrics2022

Capacity exhaustion1 case

Component
DatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Untested backupUnknown limitLimited debug accessMonitoring blind spotNo runbook
Organisation
Instapaper
  1. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017

Software defect1 case

Component
DatabaseDatabase replicationDatabase failover manager
Contributing factor
Automation amplificationHarmful manual actionOverlapping failuresDependence on a warm cache
Organisation
GitHub
  1. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went down
    GitHub2012