Database replication

Cases, by severity

1. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hoursLootLocker · 2026 · sev2

   Root cause Certificate or credential expiry

2. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026

   Root cause Capacity exhaustion

3. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026

   Root cause Breaking change in an upgrade

4. IL-0288 ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026

   Root cause Capacity exhaustion

5. IL-0281 PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026

   Root cause Software defect

6. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026

   Root cause Capacity exhaustion

7. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outagePlanetScale · 2025

   Root cause Cloud provider outage

8. IL-0242 Both PostgreSQL servers behind matrix.org were lost during a disk expansion, and the homeserver was down for 24 hoursMatrix.org Foundation · 2025

   Root cause Operator error

9. IL-0527 OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025

   Root cause Software defect

10. IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databasesUpstash · 2025

    Root cause Software defect

11. IL-0526 OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinksOpenStreetMap Foundation · 2024

    Root cause Dependency failure

12. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024

    Root cause Configuration change

13. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorumDigitalOcean · 2024

    Root cause Configuration change

14. IL-0488 Kit's RDS MySQL write latency rose 100x and running threads jumped from 9 to 900, backing up Sidekiq job queuesKit · 2022

    Root cause Cloud provider outage

15. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slowRevenueCat · 2022

    Root cause Breaking change in an upgrade

16. IL-0311 BattleMetrics' primary PostgreSQL went read-only at transaction ID wraparound, breaking the website and RCON updatesBattleMetrics · 2022

    Root cause Data loss or corruption

17. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicasCloudflare · 2020

    Root cause DNS or network fault

18. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020

    Root cause Dependency failure

19. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020

    Root cause Configuration change

20. IL-0443 A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019

    Root cause Capacity exhaustion

21. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-onlyWikimedia Foundation · 2019

    Root cause Hardware failure

22. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hoursGitHub · 2018

    Root cause DNS or network fault

23. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degradedHatena · 2018

    Root cause Configuration change

24. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017

    Root cause Operator error

25. IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deletedDigitalOcean · 2017

    Root cause Configuration change

26. IL-0334 Instapaper's RDS MySQL bookmarks table hit the 2TB ext3 file size limit, inserts failed and the service went downInstapaper · 2017

    Root cause Capacity exhaustion

27. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017

    Root cause Operator error

28. IL-0296 Transloadit's us-east Redis cluster lost queued jobs and heartbeats, raising WORKER_JOB_ERROR errors for 81 customersTransloadit · 2016

    Root cause Capacity exhaustion

29. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015

    Root cause Operator error

30. IL-0590 F5 load balancers cached a path MTU of 296 after Rackspace reboots, and Mailgun connections failed intermittentlyMailgun · 2014

    Root cause DNS or network fault

31. IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedlyTwilio · 2013

    Root cause Configuration change

32. IL-0589 Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013

    Root cause Software defect

33. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012

    Root cause Software defect