Incident Library
Applied filters ComponentDatabase proxy and connection poolerSymptomDatabase overloadClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022
    Symptoms

    On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

    Writes rejectedDatabase overloadHigh error rateFailover failure
    Trigger

    Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius
    Remediation
    • Failed over mysql1 to a healthy replica on March 16
    • Reset the application connections to the new primary on March 17
    • Added an index that fixed the main query performance problem
    • Performed a primary failover on March 22
    • Performed a primary failover on March 23
    • Throttled webhook traffic to reduce load
    Action items
    • Keep throttling webhook traffic at peak load times
    • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
    • Move traffic to other databases to reduce load and speed up failover
    • Review change management, particularly monitoring and changes during high load in production
    • Shard the databases and scale up hardware
    View case
Check your cluster for this

These 6 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Configuration change1 case

Component
DatabaseDatabase proxy and connection pooler
Contributing factor
No staged rolloutRetry stormUnrepresentative staging
Organisation
Safie
  1. IL-0574 Aurora MySQL writer CPU stuck near 100% minutes after Safie pointed production at a new Amazon RDS Proxy
    Safie2026

Faulty deployment2 cases

Component
Database2Database proxy and connection pooler2Liveness and readiness probes1Schema migration1
Contributing factor
Limited debug access1No configuration tests1Health check tied to a dependency1
Organisation
Railway2
  1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
  2. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025

Breaking change in an upgrade2 cases

Component
Database2Database proxy and connection pooler2PostgreSQL vacuum1Database replication1Query planner and table statistics1
Contributing factor
Monitoring blind spot2Unverified assumption1Automation amplification1Harmful manual action1Overlapping failures1Retry storm1Unrepresentative staging1Unpinned version1
Organisation
Clerk1RevenueCat1
  1. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
  2. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow
    RevenueCat2022

Capacity exhaustion1 case

Component
DatabaseDatabase proxy and connection pooler
Contributing factor
Excessive blast radiusChange during peak hoursInefficient queryHarmful manual action
Organisation
GitHub
  1. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022