Cases2 cases

1 to 2 of 2

1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards

   Intercom2026

   Symptoms

   From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

   Service outage

   Trigger

   At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionOverlapping failuresAutomation amplification

   Detection

   Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

   Remediation

   • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
   • Removed the artifacts left behind by the failed MoveTables workflow
   • Rolled Vitess back to v21

   Action items

   • PlanetScale to deploy a fix for the Vitess v22 regression
   • Pause further database upgrades for Intercom until the patch is verified
   • Find the source of the packet drops in the provider's Kubernetes networking
   • Explore safeguards that protect the application from invalid topology updates
   • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
   • Separate maintenance tasks from version upgrades more strictly so every deployment is intended

   View case
2. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days

   Clerk2025

   Symptoms

   At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

   High error rateHigh latencyDatabase overloadConnection pool exhaustion

   Trigger

   At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action

   Detection

   Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

   Remediation

   • Optimised several expensive queries, refined critical indexes and re-indexed the database
   • Blocked abusive traffic more aggressively for a time
   • Performed a manual minor version upgrade and increased database capacity
   • Added jitter to the connection pooling configuration so connections no longer recycled in step

   Action items

   • Evaluate other database providers that give more control over upgrades and downgrades
   • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
   • Keep optimising the most costly database queries
   • Evaluate sharding and multi-tenant infrastructure to isolate one application from others

   Lessons

   Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

   View case

By root cause1