Cases2 cases

1 to 2 of 2

1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards

   Intercom2026

   Symptoms

   From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

   Service outage

   Trigger

   At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionOverlapping failuresAutomation amplification

   Detection

   Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

   Remediation

   • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
   • Removed the artifacts left behind by the failed MoveTables workflow
   • Rolled Vitess back to v21

   Action items

   • PlanetScale to deploy a fix for the Vitess v22 regression
   • Pause further database upgrades for Intercom until the patch is verified
   • Find the source of the packet drops in the provider's Kubernetes networking
   • Explore safeguards that protect the application from invalid topology updates
   • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
   • Separate maintenance tasks from version upgrades more strictly so every deployment is intended

   View case
2. IL-0265 Open Build Service returned 503 errors for about two hours after an HAProxy package update changed its configuration

   Open Build Service2023

   Symptoms

   On 5 January 2023 the Open Build Service (OBS) reference server went offline for about two hours, one minute after a package update started at 10:26 UTC. The application answered with a maintenance message or an HTTP 503 error (Service Unavailable), and nobody could work with the API or the web interface.

   Service outage

   Trigger

   The haproxy package on the reference server was updated with zypper; the update started after the Thursday maintenance window had ended, because the window was wrongly assumed to end at 11AM.

   Root cause

   Breaking change in an upgrade

   Component

   HAProxy

   Contributing factor

   Overlapping failuresMonitoring blind spotConfiguration not in code

   Detection

   Both the team and customers saw that the application was down right after the update; customers reported it through different communication channels.

   Remediation

   • Adjusted all the HAProxy configuration files
   • Removed the maintenance mode flag from the Apache configuration
   • Restarted the server

   Action items

   • Discuss whether a maintenance configuration block is a benefit or a problem
   • Review the synchronization of SSL certificates for the secondary proxies

   Lessons

   The OBS team wants to analyse the consequences of running "zypper up" more carefully, because some upgrades need attention. Secondary machines and their configuration need watching too: although everything is usually deployed with Salt, the certificates for the secondary machines were missed. When Passenger is started through Apache, it will not start if Apache has problems, so the whole Apache configuration should be checked even when the daemon starts without visible errors. The team also noted to expect the unexpected after a long vacation break.

   View case

By root cause1