Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
    Symptoms

    From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

    Service outage
    Trigger

    At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionOverlapping failuresAutomation amplification
    Detection

    Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

    Remediation
    • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
    • Removed the artifacts left behind by the failed MoveTables workflow
    • Rolled Vitess back to v21
    Action items
    • PlanetScale to deploy a fix for the Vitess v22 regression
    • Pause further database upgrades for Intercom until the patch is verified
    • Find the source of the packet drops in the provider's Kubernetes networking
    • Explore safeguards that protect the application from invalid topology updates
    • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
    • Separate maintenance tasks from version upgrades more strictly so every deployment is intended
    View case
  2. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
    Symptoms

    At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

    High error rateHigh latencyDatabase overloadConnection pool exhaustion
    Trigger

    At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action
    Detection

    Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

    Remediation
    • Optimised several expensive queries, refined critical indexes and re-indexed the database
    • Blocked abusive traffic more aggressively for a time
    • Performed a manual minor version upgrade and increased database capacity
    • Added jitter to the connection pooling configuration so connections no longer recycled in step
    Action items
    • Evaluate other database providers that give more control over upgrades and downgrades
    • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
    • Keep optimising the most costly database queries
    • Evaluate sharding and multi-tenant infrastructure to isolate one application from others
    Lessons

    Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

    View case
  3. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers
    CircleCI2021
    Symptoms

    From 13:30 UTC on 21 May 2021, CircleCI jobs on Docker, the Machine executor (including Windows, Mac, Arm and GPU) and remote Docker were delayed or blocked. Engineers had been paged at 13:00 for excessive delays in starting jobs, and the alerts showed CircleCI could not provision some types and classes of VM because it was hitting its CPU quotas in GCP.

    Throughput degradationCloud API rate limit exhaustedDatabase overload
    Trigger

    The upgrade of the RabbitMQ cluster from 3.8.9 to 3.8.16 on 20 May 2021, which brought in a 15 minute consumer acknowledgement timeout for all queue types.

    Root cause
    Breaking change in an upgrade
    Component
    Message brokerDatabase
    Contributing factor
    Change with a delayed effectIgnored early signalRetry stormExcessive blast radiusOverlapping failures
    Detection

    Engineers were paged at 13:00 UTC on 21 May for excessive delays in starting jobs. Earlier, a 02:31 notification that a VM destroyer queue had no consumers was handled by restarting pods, and a 10:11 alert that a VM destroy queue was backing up notified once at its threshold and did not page.

    Remediation
    • Blocked Linux machine jobs from entering the work queue
    • Had GCP raise the CPU quotas
    • Deleted old VMs by hand
    • Cancelled some workflows to free resources
    • Reset the VM services deployment so the VM destroy queues had consumers again
    • Scaled up the RDS database instance, at 18:20 and again at 19:15
    • Turned down VM provisioning and blocked remote-docker jobs to shed load
    • Scaled down all pods that created and destroyed VMs
    • Deployed a revised distribution backend that kept remote-docker jobs out of the provisioning queues
    • Brought provisioning and distribution back one component at a time and reversed the mitigation code changes
    Action items
    • Make VM provisioning more robust to changes in load so it can recover from thundering herds
    • Make VM provisioning independent so its failure does not affect Docker jobs
    • Review and improve observability of RabbitMQ and the production queues
    • Batch requests to avoid hitting GCP API rate limits
    • Improve incident response for long-running incidents
    View case
Check your cluster for this

These 3 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade3 cases

Component
Database3Database proxy and connection pooler2Message broker1
Contributing factor
Overlapping failures3Retry storm2Unpinned version2Automation amplification1Excessive blast radius1Change with a delayed effect1Ignored early signal1Harmful manual action1Monitoring blind spot1
Organisation
CircleCI1Clerk1Intercom1
  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
  2. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
  3. IL-0294 CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumers
    CircleCI2021