Cases1 case

1 to 1 of 1

1. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data

   Intercom2026

   Symptoms

   From 17:03 UTC on January 27, 2026, a small subset of Intercom customers, fewer than 1%, saw inconsistent latency and stale data when reading from the platform. At 17:25 UTC an alarm on an excessive message backlog in a worker declared an incident automatically.

   Replication lagDatabase overloadHigh latency

   Trigger

   An internal app deletion process, running more concurrent deletion steps per app than usual with no rate limiting, and the ETL data pipeline both put heavy load on the same database shard at the same time.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   No headroom

   Detection

   An alarm on an excessive message backlog in a worker fired at 17:25 UTC and declared an incident automatically.

   Remediation

   • Disabled ETL processing to reduce load on the affected shard
   • Scaled the app deletion workers to zero
   • Re-enabled ETL carefully once monitoring showed stable metrics

   Action items

   • Increase the capacity of shards to leave headroom for unexpected load spikes
   • Add rate limiting and resource controls to the app deletion process so it backs off when database load is high
   • Review how background work such as ETL and maintenance tasks competes with customer-facing operations for database capacity
   • Alert on replication lag earlier and link it to specific workloads
   • Communicate proactively with affected customers in incidents with a limited blast radius

   Lessons

   Because data is spread across many independent shards, the overload stayed on one shard and reached fewer than 1% of customers instead of the whole platform.

   View case

By root cause1