Cases3 cases

1 to 3 of 3

1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards

   Intercom2026

   Symptoms

   From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

   Service outage

   Trigger

   At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionOverlapping failuresAutomation amplification

   Detection

   Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

   Remediation

   • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
   • Removed the artifacts left behind by the failed MoveTables workflow
   • Rolled Vitess back to v21

   Action items

   • PlanetScale to deploy a fix for the Vitess v22 regression
   • Pause further database upgrades for Intercom until the patch is verified
   • Find the source of the packet drops in the provider's Kubernetes networking
   • Explore safeguards that protect the application from invalid topology updates
   • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
   • Separate maintenance tasks from version upgrades more strictly so every deployment is intended

   View case
2. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking

   Heroku2025

   Symptoms

   From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

   Network unreachableService outageHigh error rate

   Trigger

   An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

   Root cause

   Breaking change in an upgrade

   Component

   Host routes and routing rules

   Contributing factor

   Unpinned versionCircular dependencyLimited debug access

   Detection

   Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

   Remediation

   • Restarted internal instances in rotation, which gave some relief
   • Had an upstream vendor invalidate the token used for the automated updates
   • Disabled the automated upgrade service
   • Recycled dynos across the whole fleet

   Action items

   • Put permanent controls in place so no system change happens outside the controlled deployment process
   • Audit all base images for similar risks
   • Make network routing survive graceful restarts of the networking service
   • Build fully independent backup communication channels
   • Overhaul incident response tooling, including tools that run queries across the whole fleet
   • Streamline break-glass procedures for rapid access to critical systems
   • Improve monitoring to detect complex issues earlier

   Lessons

   Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

   View case
3. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java

   PortSwigger2024

   Symptoms

   On 24 October 2024 every BSEE SaaS customer instance showed the Environment Unavailable message, and the SaaS Uptime Over Time metric dropped sharply at 04:45. The Technology Team learned of the outage when the Growth team posted about it at 08:07 and a member of the Product team told the Technology Team verbally at 08:15.

   Container crash loopService outage

   Trigger

   AWS made Bottlerocket v1.26.0 the latest Bottlerocket AMI in eu-west-1 between 02:00 and 04:30, and at 04:00 Karpenter saw the new version and started replacing the EC2 nodes of the production EKS cluster with it.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageKarpenter

   Contributing factor

   Unpinned version

   Detection

   The Growth team reported the outage in the BSEE cloud onboarding issues channel at 08:07, and a member of the Product team told the Technology Team verbally at 08:15. The SaaS Uptime Over Time metric had already dropped at 04:45.

   Remediation

   • Reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0
   • Deployed Bottlerocket 1.25.0 across all EKS clusters and rolled out new EKS nodes

   Action items

   • Change the Karpenter resources to deploy a specific versioned Bottlerocket AMI instead of the latest released AMI
   • Add the Bottlerocket release ATOM feed to an internal tooling release dashboard
   • Set up a public status page to communicate with customers

   View case

By root cause1