Incident Library
Applied filters Root causeBreaking change in an upgradeOrganisationIntercomClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
    Symptoms

    From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

    Service outage
    Trigger

    At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionOverlapping failuresAutomation amplification
    Detection

    Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

    Remediation
    • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
    • Removed the artifacts left behind by the failed MoveTables workflow
    • Rolled Vitess back to v21
    Action items
    • PlanetScale to deploy a fix for the Vitess v22 regression
    • Pause further database upgrades for Intercom until the patch is verified
    • Find the source of the packet drops in the provider's Kubernetes networking
    • Explore safeguards that protect the application from invalid topology updates
    • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
    • Separate maintenance tasks from version upgrades more strictly so every deployment is intended
    View case
Check your cluster for this

This case involves Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade1 case

Component
DatabaseDatabase proxy and connection pooler
Contributing factor
Automation amplificationOverlapping failuresUnpinned version
Organisation
Intercom
  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026