Incident Library

Symptom

Replication lag

복제 지연 · replication-lag

18 cases · 2012 to 2026

Root cause Capacity exhaustion4Operator error3Software defect3Cloud provider outage2Configuration change2Dependency failure1DNS or network fault1Hardware failure1Breaking change in an upgrade1
Component Database17Database replication17Database backups and snapshots4Database failover manager4Cloud block storage2Backbone network (WAN)1BGP routing1In-memory cache1Coordination and key-value store1Database proxy and connection pooler1Liveness and readiness probes1NIC, cable and switch port link1Object storage1Pod scheduling and placement1Telemetry agent1
Contributing factor Automation amplification4Monitoring blind spot4Unverified assumption3Unreviewed defaults3Harmful manual action3Pages not delivered3Single point of failure3Untested backup2No runbook2Overlapping failures2Signal buried in noise1Change with a delayed effect1Ignored early signal1No alert on limit1No headroom1Retry storm1Unrepresentative staging1Gap in automated tests1Dependence on a warm cache1
Organisation GitHub2GitLab2Wikimedia Foundation237signals1CDN771DeNA1Hatena1InfluxData1Intercom1Kickstarter1OpenStreetMap Foundation1Oregon State University Open Source Lab1PlanetScale1Razorpay1Repro1

Cases

  1. IL-0379 InfluxDB Cloud queries in Azure West Europe slowed and timed out as storage pods fell behind and kept restartingInfluxData · 2026

    Root cause Capacity exhaustion

  2. IL-0322 A GCP storage incident degraded the disk under GitLab.com's primary PostgreSQL and caused two bursts of site-wide 503sGitLab · 2026 · sev1

    Root cause Cloud provider outage

  3. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026

    Root cause Breaking change in an upgrade

  4. IL-0288 ZooKeeperWait rose to thousands of seconds on three redundant ClickHouse instances, degrading a CDN77 data pipelineCDN77 · 2026

    Root cause Capacity exhaustion

  5. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox dataIntercom · 2026

    Root cause Capacity exhaustion

  6. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outagePlanetScale · 2025

    Root cause Cloud provider outage

  7. IL-0527 OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025

    Root cause Software defect

  8. IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024

    Root cause Configuration change

  9. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020

    Root cause Dependency failure

  10. IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020

    Root cause Configuration change

  11. IL-0443 A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or moreRepro · 2019

    Root cause Capacity exhaustion

  12. IL-0323 A RAID battery backup unit failure crashed db1075, the MariaDB primary of section s3, and every s3 wiki went read-onlyWikimedia Foundation · 2019

    Root cause Hardware failure

  13. IL-0210 GitHub's MySQL clusters failed over to the US West Coast and the site served slow, stale, inconsistent data for 24 hoursGitHub · 2018

    Root cause DNS or network fault

  14. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017

    Root cause Operator error

  15. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017

    Root cause Operator error

  16. IL-0319 Dropping databases on the MySQL replica mysql2 also dropped them on mysql1, as the pair actually ran master-masterOregon State University Open Source Lab · 2015

    Root cause Operator error

  17. IL-0589 Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013

    Root cause Software defect

  18. IL-0303 GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012

    Root cause Software defect

View on the graph