Incident Library
Applied filters Root causeBreaking change in an upgradeComponentDatabase proxy and connection poolerFactorUnpinned versionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
    Symptoms

    From 19:39 to 20:50 UTC on January 9, 2026, all Intercom services in the US region were unavailable to customers hosted there. Queries from the Intercom application could not reach the database shards, and Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC.

    Service outage
    Trigger

    At 18:56 UTC a configuration in the database management pipeline treated a 'branch unfreeze' action as a signal to start a Vitess v22 upgrade planned for the following week. At 19:30 UTC a MoveTables workflow failed during transient networking issues in the provider's Kubernetes cluster, and the v22 rollback logic applied an empty VSchema.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionOverlapping failuresAutomation amplification
    Detection

    Inbox Heartbeat Anomaly alarms fired in the US region at 19:40 UTC and declared an incident; engineers and an incident commander acknowledged it at 19:41 UTC.

    Remediation
    • PlanetScale engineers reapplied a VSchema snapshot from one hour before the incident
    • Removed the artifacts left behind by the failed MoveTables workflow
    • Rolled Vitess back to v21
    Action items
    • PlanetScale to deploy a fix for the Vitess v22 regression
    • Pause further database upgrades for Intercom until the patch is verified
    • Find the source of the packet drops in the provider's Kubernetes networking
    • Explore safeguards that protect the application from invalid topology updates
    • Review the pipeline configuration that let a branch unfreeze trigger a pending version upgrade
    • Separate maintenance tasks from version upgrades more strictly so every deployment is intended
    View case
  2. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory
    Railway2025
    Symptoms

    At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.

    High latencyIntermittent timeoutsDeploys failing
    Trigger

    After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionNo staged rolloutMonitoring blind spotUnreviewed defaults
    Detection

    Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.

    Remediation
    • Deployed a fix for the API failure caused by the schema change within 15 minutes
    • Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
    • Suspended all deployments, Pro included, for up to 20 minutes during peak instability
    • Scaled up the control plane database's CPU, memory, connection limits and IOPS
    • Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
    • Re-enabled deployments for all non-Pro users
    • Fixed the edge case in which private networks and their endpoints were missing from the dashboard
    Action items
    • Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
    • Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
    • Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
    • Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once
    Lessons

    Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.

    View case
  3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
    Symptoms

    At 13:08 UTC on 15 September 2025, Clerk's internal monitoring alerted on significant request failures and increased latency. Over the following days the request failures and latency spikes kept coming back while the Postgres database stayed unhealthy, with repeated spikes in database load.

    High error rateHigh latencyDatabase overloadConnection pool exhaustion
    Trigger

    At 05:30 UTC on 14 September 2025 the cloud provider automatically upgraded Clerk's Postgres database to a new minor version, which changed how Postgres handled connection locks.

    Root cause
    Breaking change in an upgrade
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Unpinned versionRetry stormMonitoring blind spotOverlapping failuresHarmful manual action
    Detection

    Internal monitoring alerted on significant request failures and increased latency at 13:08 UTC on 15 September. Shortly after the upgrade on 14 September it had already shown an increased but still acceptable database load.

    Remediation
    • Optimised several expensive queries, refined critical indexes and re-indexed the database
    • Blocked abusive traffic more aggressively for a time
    • Performed a manual minor version upgrade and increased database capacity
    • Added jitter to the connection pooling configuration so connections no longer recycled in step
    Action items
    • Evaluate other database providers that give more control over upgrades and downgrades
    • Isolate more services from each other, for example Sign Up and Sign In infrastructure from the rest of the system
    • Keep optimising the most costly database queries
    • Evaluate sharding and multi-tenant infrastructure to isolate one application from others
    Lessons

    Clerk's review found that connection metrics sampled every 60 seconds could not show a recycling pattern that lasted seconds and repeated every 15 minutes, that an attack and background jobs with similar periods pulled attention away from the database layer, that the Backend API looked healthy only because it manages far fewer connections, and that the release freeze removed the staggering of connection creation that normal deploys provide.

    View case
Check your cluster for this

These 3 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade3 cases

Component
Database3Database proxy and connection pooler3
Contributing factor
Unpinned version3Monitoring blind spot2Overlapping failures2Automation amplification1Unreviewed defaults1Harmful manual action1No staged rollout1Retry storm1
Organisation
Clerk1Intercom1Railway1
  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
  2. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory
    Railway2025
  3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025