IL-0304 Intercom · 2026-01-09database
Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
Intercom uses PlanetScale, built on Vitess, as its core database layer. At 18:56 UTC on January 9, 2026, a configuration in the database management pipeline read a 'branch unfreeze' action as a signal to go ahead with a pending upgrade to Vitess v22 that was scheduled for the following week, and the upgrade began rolling out at 19:00 UTC. At 19:30 UTC a MoveTables workflow, part of a standard administrative task to move tables for testing, failed because of transient networking issues in the provider's Kubernetes cluster.
When a MoveTables command fails, Vitess attempts to revert its state. For unsharded keyspaces it keeps the previous VSchema in memory for this rollback, and for sharded keyspaces it saves nothing, because the operation is not expected to change the VSchema there. A regression in v22, introduced in Vitess Pull Request #17401, made the check for whether to reapply the old VSchema come out true for Intercom's sharded keyspace, so Vitess applied a previous VSchema it had never saved, which was empty. At 19:39 UTC the PlanetScale control plane read the empty VSchema and passed it to the VTGates, which dropped all routes.
Engineers and an incident commander acknowledged the incident at 19:41 UTC and engaged PlanetScale at 19:48 UTC. At 20:50 UTC PlanetScale engineers reapplied a VSchema snapshot taken one hour before the incident and removed what the failed workflow had left behind; the control plane pushed the valid routing rules to the VTGates, which rebuilt their routing tables, and service recovered. From 21:00 UTC Vitess was rolled back to v21 over the following hours. Intercom found no evidence of data loss or corruption. The networking packet drops in the provider's Kubernetes cluster may have contributed by causing the workflow failure that set off the bug.
- Severity
- –
- Scope
- All Intercom services in the US region were unavailable for 71 minutes
- Users affected
- Intercom customers hosted in the US region
- Region
- US
- Time to detect
- 1m
- Time to resolve
- 1h 11m
Symptoms
From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.
Intercom's core database layer is PlanetScale, built on Vitess, and its proxy layer, the VTGates, routes each query to the right shard using the VSchema. An empty VSchema had been applied to the production keyspace, the PlanetScale control plane passed it to the VTGates, and the VTGates dropped all their routes, so the application lost all connectivity to its data. Queries could not be routed or run, but stored data was not changed.
Symptom Service outage
Root cause
Root cause Breaking change in an upgrade
- Trigger
- At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.
Detection
Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.
Remediation
- PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
- Removed the artifacts left behind by the failed MoveTables workflow
- Rolled Vitess back to v21
Action items
- PlanetScale to deploy a fix for the Vitess v22 regression
- Pause further database upgrades for Intercom until the patch is verified
- Find the source of the packet drops in the provider's Kubernetes networking
- Explore safeguards that protect the application from invalid topology updates
- Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
- Separate maintenance tasks from version upgrades more strictly so every deployment is intended
Lessons
–
Sources
Related cases
IL-0316Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memoryRailway · 2025same root cause · shared Database proxy and connection pooler, Database
IL-0498Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four daysClerk · 2025same root cause · shared Database proxy and connection pooler, Database
IL-0308Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slowRevenueCat · 2022same root cause · shared Database proxy and connection pooler, Database
IL-0572About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a dayMercari · 2026same root cause · shared Database
IL-0569After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026same root cause · shared Database
IL-0444Reader connections to Repro's Aurora MySQL cluster were cut with Lost connection errors every time ALTER TABLE ranRepro · 2024same root cause · shared Database
IL-0117A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024same root cause · shared Database
IL-0314Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restartingDaily · 2023same root cause · shared Database
IL-0294CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumersCircleCI · 2021same root cause · shared Database
IL-0372Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changedHeroku · 2021same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0304
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0304/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.