Incident Library
Applied filters Root causeBreaking change in an upgradeSymptomService outageFactorOverlapping failuresClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
    Symptoms

    From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

    Service outage
    Trigger

    At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionOverlapping failuresAutomation amplification
    Detection

    Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

    Remediation
    • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
    • Removed the artifacts left behind by the failed MoveTables workflow
    • Rolled Vitess back to v21
    Action items
    • PlanetScale to deploy a fix for the Vitess v22 regression
    • Pause further database upgrades for Intercom until the patch is verified
    • Find the source of the packet drops in the provider's Kubernetes networking
    • Explore safeguards that protect the application from invalid topology updates
    • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
    • Separate maintenance tasks from version upgrades more strictly so every deployment is intended
    View case
  2. IL-0265 Open Build Service returned 503 errors for about two hours after an HAProxy package update changed its configuration
    Open Build Service2023
    Symptoms

    On 5 January 2023 the Open Build Service (OBS) reference server went offline for about two hours, one minute after a package update started at 10:26 UTC. The application answered with a maintenance message or an HTTP 503 error (Service Unavailable), and nobody could work with the API or the web interface.

    Service outage
    Trigger

    The haproxy package on the reference server was updated with zypper; the update started after the Thursday maintenance window had ended, because the window was wrongly assumed to end at 11AM.

    Root cause
    Breaking change in an upgrade
    Component
    HAProxy
    Contributing factor
    Overlapping failuresMonitoring blind spotConfiguration not in code
    Detection

    Both the team and customers saw that the application was down right after the update; customers reported it through different communication channels.

    Remediation
    • Adjusted all the HAProxy configuration files
    • Removed the maintenance mode flag from the Apache configuration
    • Restarted the server
    Action items
    • Discuss whether a maintenance configuration block is a benefit or a problem
    • Review the synchronization of SSL certificates for the secondary proxies
    Lessons

    The OBS team wants to analyse the consequences of running "zypper up" more carefully, because some upgrades need attention. Secondary machines and their configuration need watching too: although everything is usually deployed with Salt, the certificates for the secondary machines were missed. When Passenger is started through Apache, it will not start if Apache has problems, so the whole Apache configuration should be checked even when the daemon starts without visible errors. The team also noted to expect the unexpected after a long vacation break.

    View case
Check your cluster for this

These 2 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
Database1Database proxy and connection pooler1HAProxy1
Contributing factor
Overlapping failures2Automation amplification1Configuration not in code1Monitoring blind spot1Unpinned version1
Organisation
Intercom1Open Build Service1
  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
  2. IL-0265 Open Build Service returned 503 errors for about two hours after an HAProxy package update changed its configuration
    Open Build Service2023