Cases7 cases

1 to 5 of 7

1. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data

   Intercom2026

   Symptoms

   From 17:03 UTC on January 27, 2026, a small subset of Intercom customers, fewer than 1%, saw inconsistent latency and stale data when reading from the platform. At 17:25 UTC an alarm on an excessive message backlog in a worker declared an incident automatically.

   Replication lagDatabase overloadHigh latency

   Trigger

   An internal app deletion process, running more concurrent deletion steps per app than usual with no rate limiting, and the ETL data pipeline both put heavy load on the same database shard at the same time.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   No headroom

   Detection

   An alarm on an excessive message backlog in a worker fired at 17:25 UTC and declared an incident automatically.

   Remediation

   • Disabled ETL processing to reduce load on the affected shard
   • Scaled the app deletion workers to zero
   • Re-enabled ETL carefully once monitoring showed stable metrics

   Action items

   • Increase the capacity of shards to leave headroom for unexpected load spikes
   • Add rate limiting and resource controls to the app deletion process so it backs off when database load is high
   • Review how background work such as ETL and maintenance tasks competes with customer-facing operations for database capacity
   • Alert on replication lag earlier and link it to specific workloads
   • Communicate proactively with affected customers in incidents with a limited blast radius

   Lessons

   Because data is spread across many independent shards, the overload stayed on one shard and reached fewer than 1% of customers instead of the whole platform.

   View case
2. IL-0488 Kit's RDS MySQL write latency rose 100x and running threads jumped from 9 to 900, backing up Sidekiq job queues

   Kit2022

   Symptoms

   At 5:20 am EST on December 24, 2022, Kit's infrastructure team got its first page because the number of MySQL threads running on its RDS MySQL primary in us-east-2 had jumped 100x, from 9 to 900, at a time of day when Kit's traffic is very low. Sidekiq queues were backing up.

   Database overloadThroughput degradation

   Trigger

   An EBS interruption on the standby of the Multi-AZ RDS instance, reported by AWS as an LSE at 02:12:47 UTC on 2022-12-24, started a replaceSecondaryHostAndVolumesTask that replaced the secondary host and volumes.

   Root cause

   Cloud provider outage

   Component

   DatabaseCloud block storageDatabase replication

   Contributing factor

   Ignored early signal

   Detection

   A page fired at 5:20 am EST because MySQL threads running jumped from 9 to 900. An AWS RSS feed alert in Slack about degraded EC2 and EBS in use2-az1 had arrived the evening before and was not acted on.

   Remediation

   • Scaled the RDS storage from 12.5 TB to 13.75 TB, which put the instance through "optimizing storage" and brought write latency back to 1-2 ms

   Lessons

   Multi-AZ RDS keeps data consistent and durable, but synchronous replication means each write also waits for the standby, so a degraded EBS volume on the standby while it is being replaced slows writes on the primary. Large databases recover more slowly from such events than small ones. A single-AZ instance would not have been hit by a standby problem, but would have suffered hours of downtime if its own zone failed, and that trade-off has to be weighed when deploying production RDS instances.

   View case
3. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow

   RevenueCat2022

   Symptoms

   At 9:34 UTC on November 23, 2022, RevenueCat started moving traffic from its Aurora PostgreSQL 10.x cluster to a new Aurora PostgreSQL 14.x cluster. By 9:39 UTC all traffic was on the new cluster, latency was bad, and the RevenueCat dashboard and the endpoints used by the RevenueCat SDK stopped working.

   Database overloadHigh latencyService outageConnection pool exhaustion

   Trigger

   At 9:34 UTC production traffic was switched from the Aurora PostgreSQL 10.x cluster to the new Aurora PostgreSQL 14.x cluster, before ANALYZE had run on its largest tables.

   Root cause

   Breaking change in an upgrade

   Component

   DatabasePostgreSQL vacuumDatabase replicationDatabase proxy and connection poolerQuery planner and table statistics

   Contributing factor

   Unrepresentative stagingUnverified assumptionAutomation amplificationMonitoring blind spot

   Detection

   At 9:39 UTC, with all traffic on the new cluster, latency looked bad and the infrastructure team began to investigate.

   Remediation

   • Added a statement timeout to the database user so queries could not hold a connection too long
   • Shed 90% of traffic by returning 503s from the load balancer
   • Dropped indexes that the planner was using wrongly and that were not needed
   • Stopped accepting Apple webhooks for a time
   • Rolled out a release without New Relic to stop its EXPLAIN queries
   • Ran ANALYZE by hand on the biggest tables
   • Raised the values of the affected primary key sequences with a prepared script
   • Stopped the integration event dispatcher by hand
   • Replayed the webhooks that were dispatched but not recorded
   • Restored integration event dispatching, ETLs and other non-web server jobs

   Action items

   • Consider change freezes during business-critical periods
   • Explain better how the SDK behaves during backend outages
   • Tell customers about maintenance and outages in the developer dashboard and by direct email to technical contacts
   • Replay a subset of write traffic in migration dry runs to mimic the cutover
   • Move costly processes to asynchronous work to keep the API fast and stable
   • Re-evaluate table indexes, including unused ones the database may choose for queries
   • Show the load of each endpoint and system in the database dashboards
   • Involve support and customer communications staff during outages

   Lessons

   The new cluster had not been analyzed, and the tests missed it because the test cluster had been up long enough to gather statistics. Dry runs replayed no write traffic, on the belief that writes are simpler than reads, and the team assumed sequence state was replicated with the data. Rolling back at the start would have been better than fixing forward. The migration, timed to finish before Thanksgiving and Black Friday, should have been delayed until the week after, because the US had entered a sensitive period and part of the US support team was already on holiday.

   View case
4. IL-0240 Cloudflare API requests failed and the dashboard slowed for hours after the authentication database lost its replicas

   Cloudflare2020

   Symptoms

   On 2 November 2020 the Cloudflare API and dashboard were degraded for six hours and 33 minutes. The success rate of API queries dropped at times to 75%, and the dashboard was up to 80 times slower than normal, while Cloudflare's edge network kept working.

   High error rateHigh latencyDatabase overloadWrites rejectedPacket loss

   Trigger

   At 14:43 UTC a network switch in a control plane data center partly failed: it kept LACP and BGP running but stopped forwarding some packets, and it recovered on its own six minutes later.

   Root cause

   DNS or network fault

   Component

   NIC, cable and switch port linkCoordination and key-value storeDatabase failover managerDatabaseDatabase replication

   Contributing factor

   Automation amplificationNo headroomHarmful manual action

   Detection

   At 14:43 UTC alerts fired because the misbehaving switch was unreachable to pings. Engineers then saw that the primary authentication database was overwhelmed.

   Remediation

   • Shed load from the primary authentication database by slowing SSL certificate pushes to the edge, emails and other features
   • Moved API traffic that could use read replicas to the secondary data center by hand
   • Sent authentication calls fully back to the primary data center when the dashboard got worse
   • Rebuilt the first database replica, which put itself back into service at 21:20 UTC and returned performance to normal
   • Ramped the turned-down services back up so asynchronous processing could catch up

   Action items

   • Remove the cluster management limitation that requires rebuilding all replicas after a primary is promoted
   • Redesign the user sessions system so dashboard traffic can be steered between data centers
   • Change the auto-remediation configuration so a database replica is not promoted so quickly
   • Tune each system involved to be more resilient to failures of its dependencies

   Lessons

   Every system in the chain had redundancy and none failed completely; each only degraded, which made the chain of events much harder to model and foresee. Automating replica promotion so that it could trigger on a minute's notice had been a point of pride, but for at least one database a fast promotion can do more harm than the failure it answers. Most general-purpose cluster managers use Raft or Paxos rather than Byzantine fault tolerant protocols, accepting a rare failure mode in exchange for a protocol that is simpler to implement correctly and debug.

   View case
5. IL-0218 GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hours

   GitLab2017

   Symptoms

   From around 19:00 UTC on 31 January 2017, load on GitLab.com's PostgreSQL database rose, which GitLab suspected was spam, and many users could not post comments on issues and merge requests. Around 23:00 UTC the replication of the PostgreSQL secondary fell behind and then failed, because WAL segments it still needed had already been removed from the primary.

   Database overloadReplication lagData lossService outage

   Trigger

   While trying to restore replication to the secondary, an engineer ran the removal of the PostgreSQL data directory on the primary, db1.cluster.gitlab.com, believing it was the secondary.

   Root cause

   Operator error

   Component

   DatabaseDatabase replicationDatabase backups and snapshotsCloud block storage

   Contributing factor

   Untested backupSingle point of failureNo runbookPages not deliveredUnverified assumptionUnreviewed defaults

   Detection

   The engineer who ran the deletion on the primary noticed the mistake and stopped it a second or two later.

   Remediation

   • Raised max_wal_senders from 3 to 32 and lowered max_connections from 8000 to 2000 so PostgreSQL would restart
   • Stopped the deletion on the primary a second or two after noticing it
   • Restored the database from the LVM snapshot taken about 6 hours before the outage by copying the PostgreSQL data directory from staging
   • Restored the webhooks from a separate database built from the snapshot and imported them as a SQL dump
   • Incremented all database sequences by 100,000 so IDs used before the outage could not be reused
   • Re-enabled GitLab.com gradually
   • Rebuilt projects created after 17:20 UTC in basic form from their Git repositories on the NFS cluster

   Action items

   • Change the shell prompt (PS1) on all hosts to show more clearly which host and environment is in use
   • Monitor backups with Prometheus
   • Set PostgreSQL's max_connections to a sane value
   • Investigate point-in-time recovery and continuous archiving for PostgreSQL
   • Take LVM snapshots of the production databases every hour
   • Take Azure disk snapshots of the production databases
   • Move staging to the ARM environment
   • Recover the production replicas
   • Test recovery of PostgreSQL database backups automatically
   • Improve the PostgreSQL replication documentation and runbooks
   • Investigate pgbarman for PostgreSQL backups
   • Investigate WAL-E for database backup and realtime replication
   • Build a streaming database restore
   • Assign an owner for data durability
   • Set up multiple secondaries and balance load among them with pgpool-II
   • Provide a tool for developers to run and revert Rails migrations on staging
   • Build owned recovery procedures for the rest of the GitLab.com infrastructure, not only the database
   • Publish a backup monitoring dashboard and add more backup data to it
   • Improve the abuse reporting and response system so removal for spam does not hard delete
   • Redact engineers' names in future public incident documents

   Lessons

   GitLab split the incident into two problems: GitLab.com went down because the primary's data directory was removed by mistake during a manual, undocumented replication rebuild, and recovery took over 18 hours because the backups had failed silently, disk snapshots were not enabled and the only copy sat on slow staging disks. The backup procedure was never tested because no one owned it. GitLab chose to focus on fast, regularly practised recovery and on making it obvious which host an engineer is on, rather than on blocking commands, since a safer rm only protects against one of many ways to lose data.

   View case

By root cause7