Incident Library
Applied filters Root causeBreaking change in an upgradeFactorUnpinned versionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases5 cases

1 to 5 of 5

  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
    Symptoms

    From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

    Service outage
    Trigger

    At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionOverlapping failuresAutomation amplification
    Detection

    Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

    Remediation
    • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
    • Removed the artifacts left behind by the failed MoveTables workflow
    • Rolled Vitess back to v21
    Action items
    • PlanetScale to deploy a fix for the Vitess v22 regression
    • Pause further database upgrades for Intercom until the patch is verified
    • Find the source of the packet drops in the provider's Kubernetes networking
    • Explore safeguards that protect the application from invalid topology updates
    • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
    • Separate maintenance tasks from version upgrades more strictly so every deployment is intended
    View case
  2. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory
    Railway2025
    Symptoms

    At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.

    High latencyIntermittent timeoutsDeploys failing
    Trigger

    After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionNo staged rolloutMonitoring blind spotUnreviewed defaults
    Detection

    Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.

    Remediation
    • Deployed a fix for the API failure caused by the schema change within 15 minutes
    • Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
    • Suspended all deployments, Pro included, for up to 20 minutes during peak instability
    • Scaled up the control plane database's CPU, memory, connection limits and IOPS
    • Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
    • Re-enabled deployments for all non-Pro users
    • Fixed the edge case in which private networks and their endpoints were missing from the dashboard
    Action items
    • Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
    • Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
    • Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
    • Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once
    Lessons

    Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.

    View case
  3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
    Symptoms

    At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

    High error rateHigh latencyDatabase overloadConnection pool exhaustion
    Trigger

    At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action
    Detection

    Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

    Remediation
    • Optimised several expensive queries, refined critical indexes and re-indexed the database
    • Blocked abusive traffic more aggressively for a time
    • Performed a manual minor version upgrade and increased database capacity
    • Added jitter to the connection pooling configuration so connections no longer recycled in step
    Action items
    • Evaluate other database providers that give more control over upgrades and downgrades
    • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
    • Keep optimising the most costly database queries
    • Evaluate sharding and multi-tenant infrastructure to isolate one application from others
    Lessons

    Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

    View case
  4. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking
    Heroku2025
    Symptoms

    From 06:00 UTC on 10 June 2025, Heroku services degraded badly: customers reported intermittent logins, applications failed, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well, which delayed communication with customers and left many of them without accurate status updates. Only at 13:58 UTC did the team find a workaround to post updates to the @herokustatus account on X.

    Network unreachableService outageHigh error rate
    Trigger

    An automated operating system update ran on Heroku's production infrastructure when it should have been disabled, and it restarted the hosts' networking services.

    Root cause
    Breaking change in an upgrade
    Component
    Host routes and routing rules
    Contributing factor
    Unpinned versionCircular dependencyLimited debug access
    Detection

    Customers reported intermittent logins, and Heroku's monitoring detected widespread errors with dyno networking. Heroku's own tools and the Heroku Status Page were affected as well.

    Remediation
    • Restarted internal instances in rotation, which gave some relief
    • Had an upstream vendor invalidate the token used for the automated updates
    • Disabled the automated upgrade service
    • Recycled dynos across the whole fleet
    Action items
    • Put permanent controls in place so no system change happens outside the controlled deployment process
    • Audit all base images for similar risks
    • Make network routing survive graceful restarts of the networking service
    • Build fully independent backup communication channels
    • Overhaul incident response tooling, including tools that run queries across the whole fleet
    • Streamline break-glass procedures for rapid access to critical systems
    • Improve monitoring to detect complex issues earlier
    Lessons

    Heroku's post-mortem names three areas: production lacked immutability controls, so an automated process could change it; communication was too slow, made worse by a status page that went down with the platform; and gaps in tooling and process made recovery take longer than it should have.

    View case
  5. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
    Symptoms

    On 24 October 2024 every BSEE SaaS customer instance showed the Environment Unavailable message, and the SaaS Uptime Over Time metric dropped sharply at 04:45. The Technology Team learned of the outage when the Growth team posted about it at 08:07 and a member of the Product team told the Technology Team verbally at 08:15.

    Container crash loopService outage
    Trigger

    AWS made Bottlerocket v1.26.0 the latest Bottlerocket AMI in eu-west-1 between 02:00 and 04:30, and at 04:00 Karpenter saw the new version and started replacing the EC2 nodes of the production EKS cluster with it.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imageKarpenter
    Contributing factor
    Unpinned version
    Detection

    The Growth team reported the outage in the BSEE cloud onboarding issues channel at 08:07, and a member of the Product team told the Technology Team verbally at 08:15. The SaaS Uptime Over Time metric had already dropped at 04:45.

    Remediation
    • Reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0
    • Deployed Bottlerocket 1.25.0 across all EKS clusters and rolled out new EKS nodes
    Action items
    • Change the Karpenter resources to deploy a specific versioned Bottlerocket AMI instead of the latest released AMI
    • Add the Bottlerocket release ATOM feed to an internal tooling release dashboard
    • Set up a public status page to communicate with customers
    View case
Check your cluster for this

These 5 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade5 cases

Component
Database3Database proxy and connection pooler3Host routes and routing rules1Karpenter1Node operating system image1
Contributing factor
Unpinned version5Monitoring blind spot2Overlapping failures2Automation amplification1Circular dependency1Unreviewed defaults1Limited debug access1Harmful manual action1No staged rollout1Retry storm1
Organisation
Clerk1Heroku1Intercom1PortSwigger1Railway1
  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
  2. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory
    Railway2025
  3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
  4. IL-0227 Most Heroku dynos in Private Spaces lost outbound connectivity after an automated OS update restarted host networking
    Heroku2025
  5. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024