Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases12 cases

11 to 12 of 12

  1. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes
    Stack Exchange2017
    Symptoms

    On January 24, 2017 at 17:53 UTC the Stack Exchange network, including Stack Overflow, went into read-only mode for about 5 minutes and then went offline for 12 minutes. It took the team 2 minutes to notice the problem, 5 minutes to locate its source and 10 minutes to restore service.

    Writes rejectedService outageFailover failureData loss
    Trigger

    At 17:53 UTC the primary SQL Server, NY-SQL02, triggered a bugcheck in the SQL Server process.

    Root cause
    unknown
    Component
    Database
    Contributing factor
    Single point of failure
    Remediation
    • Restarted the SQL service on NY-SQL02, which brought the network back online in read-only mode
    • Put the sites back into read-write mode after a sanity check of SQL health
    • Brought the network online without waiting for the memory dump to complete
    • Took NY-SQL02 out of production for testing, including a memory test, and ran the normal load on NY-SQL01
    • Brought the SQL cluster up to 2016 SP1 CU1
    View case
  2. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed
    Postmark2011
    Symptoms

    At about 8:15am EDT Postmark's primary MySQL server ran out of disk space, and email sending, the API and the web interface were delayed. Disk space alerts fired, but the space had grown much faster than usual and MySQL was already struggling when the alerts were noticed. MySQL kept running while it failed to write some data to disk, so MySQL-MMM, which normally detects problems and fails over to a healthy server, never triggered a failover; the team spent about 30 minutes trying to free logs and disk space before failing over by hand.

    Node disk fullWrites rejectedFailover failure
    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase failover manager
    Contributing factor
    –
    Detection

    Per-server monitoring raised disk space alerts, but the disk filled so quickly that MySQL was already struggling by the time the alerts were noticed.

    Remediation
    • Tried to free up logs and disk space on the primary server
    • Failed over manually to the second MySQL server
    • Recovered the jobs that had failed
    • Sent the emails that each server had queued locally while the database was unavailable
    Action items
    • Upgrade the MySQL cluster with a focus on availability and performance
    • Make the alerts more sensitive to catch problems sooner
    Lessons

    Looking back, Postmark said it should have failed over to the second server right away instead of first making sure replication was healthy.

    View case
Check your cluster for this

These 12 cases involve Database and Database replication. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Software defect2 cases

Component
Database2Database failover manager2
Contributing factor
Unverified assumption1Excessive blast radius1Change with a delayed effect1Harmful manual action1Monitoring blind spot1Overlapping failures1Unrepresentative staging1
Organisation
Hightouch1Stripe1
  1. IL-0310 An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactions
    Hightouch2025
  2. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day
    Stripe2019

Capacity exhaustion6 cases

Component
Database6PostgreSQL vacuum1Database backups and snapshots1Database proxy and connection pooler1Database replication1Database failover manager1
Contributing factor
Monitoring blind spot3Inefficient query2Signal buried in noise1Untested backup1Excessive blast radius1Change during peak hours1Ignored early signal1Unknown limit1Limited debug access1Harmful manual action1No alert on limit1No capacity planning1No runbook1No headroom1Retry storm1
Organisation
37signals1GitHub1Instapaper1Mailchimp1MailPace1Postmark1
  1. IL-0326 A retention clean-up job filled the disk of MailPace's managed Postgres, which stayed offline for just over two hours
    MailPace2023
  2. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022
  3. IL-0211 A Mandrill PostgreSQL shard shut down at transaction ID wraparound and email sending was impaired for about 40 hours
    Mailchimp2019
  4. IL-0300 Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,647
    37signals2018
  5. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went down
    Instapaper2017
  6. IL-0320 Postmark's primary MySQL disk filled but MySQL kept running, so MySQL-MMM never failed over and sending was delayed
    Postmark2011

Data loss or corruption1 case

Component
PostgreSQL vacuumDatabaseDatabase backups and snapshotsDatabase replication
Contributing factor
Monitoring blind spotNo headroomOverlapping failures
Organisation
BattleMetrics
  1. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updates
    BattleMetrics2022

DNS or network fault1 case

Component
Coordination and key-value storeDatabaseDatabase replicationDatabase failover managerNIC, cable and switch port link
Contributing factor
Automation amplificationHarmful manual actionNo headroom
Organisation
Cloudflare
  1. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas
    Cloudflare2020

Hardware failure1 case

Component
DatabaseDatabase replication
Contributing factor
Harmful manual actionPages not delivered
Organisation
Wikimedia Foundation
  1. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-only
    Wikimedia Foundation2019

unknown1 case

Component
Database
Contributing factor
Single point of failure
Organisation
Stack Exchange
  1. IL-0335 A bugcheck in the primary SQL Server put the Stack Exchange network into read-only mode and then offline for 12 minutes
    Stack Exchange2017