Cases2 cases

1 to 2 of 2

1. IL-0305 Replicas of one Intercom Vitess shard hit 100% CPU and fell up to 2.2 hours behind, serving stale inbox data

   Intercom2026

   Symptoms

   From 17:03 UTC on January 27, 2026, a small subset of Intercom customers, fewer than 1%, saw inconsistent latency and stale data when reading from the platform. At 17:25 UTC an alarm on an excessive message backlog in a worker declared an incident automatically.

   Replication lagDatabase overloadHigh latency

   Trigger

   An internal app deletion process, running more concurrent deletion steps per app than usual with no rate limiting, and the ETL data pipeline both put heavy load on the same database shard at the same time.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   No headroom

   Detection

   An alarm on an excessive message backlog in a worker fired at 17:25 UTC and declared an incident automatically.

   Remediation

   • Disabled ETL processing to reduce load on the affected shard
   • Scaled the app deletion workers to zero
   • Re-enabled ETL carefully once monitoring showed stable metrics

   Action items

   • Increase the capacity of shards to leave headroom for unexpected load spikes
   • Add rate limiting and resource controls to the app deletion process so it backs off when database load is high
   • Review how background work such as ETL and maintenance tasks competes with customer-facing operations for database capacity
   • Alert on replication lag earlier and link it to specific workloads
   • Communicate proactively with affected customers in incidents with a limited blast radius

   Lessons

   Because data is spread across many independent shards, the overload stayed on one shard and reached fewer than 1% of customers instead of the whole platform.

   View case
2. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards

   Intercom2026

   Symptoms

   From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

   Service outage

   Trigger

   At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionOverlapping failuresAutomation amplification

   Detection

   Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

   Remediation

   • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
   • Removed the artifacts left behind by the failed MoveTables workflow
   • Rolled Vitess back to v21

   Action items

   • PlanetScale to deploy a fix for the Vitess v22 regression
   • Pause further database upgrades for Intercom until the patch is verified
   • Find the source of the packet drops in the provider's Kubernetes networking
   • Explore safeguards that protect the application from invalid topology updates
   • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
   • Separate maintenance tasks from version upgrades more strictly so every deployment is intended

   View case

By root cause2