Incident Library
Applied filters ComponentDatabase proxy and connection poolerClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

13 to 13 of 13

  1. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022
    Symptoms

    On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

    Writes rejectedDatabase overloadHigh error rateFailover failure
    Trigger

    Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius
    Remediation
    • Failed over mysql1 to a healthy replica on March 16
    • Reset the application connections to the new primary on March 17
    • Added an index that fixed the main query performance problem
    • Performed a primary failover on March 22
    • Performed a primary failover on March 23
    • Throttled webhook traffic to reduce load
    Action items
    • Keep throttling webhook traffic at peak load times
    • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
    • Move traffic to other databases to reduce load and speed up failover
    • Review change management, particularly monitoring and changes during high load in production
    • Shard the databases and scale up hardware
    View case
Check your cluster for this

These 13 cases involve Database proxy and connection pooler and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change2 cases

Component
Database proxy and connection pooler2Coordination and key-value store1Database1Pod termination1
Contributing factor
Retry storm2Unknown limit1No staged rollout1Unrepresentative staging1
Organisation
Discord1Safie1
  1. IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS Proxy
    Safie2026
  2. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026

Software defect1 case

Component
DatabaseDatabase proxy and connection pooler
Contributing factor
Unverified assumptionUnrepresentative staging
Organisation
Decagon
  1. IL-0313 Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy pool
    Decagon2026

Breaking change in an upgrade4 cases

Component
Database4Database proxy and connection pooler4PostgreSQL vacuum1Database replication1Query planner and table statistics1
Contributing factor
Monitoring blind spot3Unpinned version3Automation amplification2Overlapping failures2Unverified assumption1Unreviewed defaults1Harmful manual action1No staged rollout1Retry storm1Unrepresentative staging1
Organisation
Clerk1Intercom1Railway1RevenueCat1
  1. IL-0304 Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shards
    Intercom2026
  2. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory
    Railway2025
  3. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
  4. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow
    RevenueCat2022

Faulty deployment3 cases

Component
Database3Database proxy and connection pooler3Liveness and readiness probes1Schema migration1
Contributing factor
Limited debug access1No configuration tests1Health check tied to a dependency1
Organisation
Railway2Supabase1
  1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
  2. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
  3. IL-0307 Supabase's Postgres connection pooler Supavisor 2.0 dropped connections for about 80% of us-east-1 tenants
    Supabase2025

Cloud provider outage1 case

Component
DatabaseDatabase proxy and connection poolerDatabase replicationObject storage
Contributing factor
–
Organisation
PlanetScale
  1. IL-0368 PlanetScale's control plane went down and some database branches failed queries during the AWS us-east-1 outage
    PlanetScale2025

Capacity exhaustion2 cases

Component
Database2Database proxy and connection pooler2Linux kernel1
Contributing factor
Excessive blast radius2Unverified assumption1Automation amplification1Change during peak hours1Inefficient query1Harmful manual action1Unrepresentative staging1
Organisation
Gadget1GitHub1
  1. IL-0312 Gadget's main Postgres did not start again after a restart to raise max_connections, and the platform went down
    Gadget2024
  2. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022