Incident Library
Applied filters Root causeCapacity exhaustionSymptomConnection pool exhaustionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed
    Buttondown2026
    Symptoms

    On 31 March 2026 Buttondown was down twice, for about seven and about thirteen minutes. New requests could not get a database connection and failed, although the database itself was healthy: queries were fast, CPU and memory were fine and replication lag was nominal.

    Connection pool exhaustionService outage
    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planningMonitoring blind spotCircular dependency
    Detection

    User reports and the team's own manual observation, not automated monitoring. The health check endpoint kept returning 200 for most requests, so no alert fired.

    Remediation
    • Killed active queries to free database connections in the first outage
    • Restarted the database to force-kill the running queries in the second outage
    Action items
    • Reserve a small number of database connections for administrative access
    • Monitor and alert on the database connection count as a percentage of the configured maximum
    • Make the health check endpoint run a lightweight query so connection exhaustion shows in the health checker and the load balancer
    • Add database connection limits to the quarterly capacity review
    • Document and script a fallback that force-recycles workers to release connections without needing a database connection
    Lessons

    Buttondown notes that the tool used to recover from the first outage needed the same resource that was exhausted, which locked the team out during the second outage, and that a health check which cannot detect connection exhaustion, the most common failure mode, gives little protection. A configuration value set long ago drifted out of line with traffic because nobody reviewed it.

    View case
  2. IL-0557 Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hours
    Storj2025
    Symptoms

    From 21:04 UTC on 7 August 2025, database transaction locks on the Storj US1 satellite started to increase, and upload and download requests began to fail. The on-call team was paged at 21:14 UTC.

    Lock contention or deadlockIntermittent timeoutsConnection pool exhaustionHigh error rate
    Trigger

    An unusually high volume of concurrent uploads to the same objects reached the US1 satellite.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    –
    Detection

    The on-call team was paged at 21:14 UTC, ten minutes after database transaction locks began to rise.

    Remediation
    • Reset the state of connections periodically to keep contention errors from growing in a cascade
    Action items
    • Remove the sources of contention between database transactions with code changes
    • Add monitoring and alerts for increased levels of database locks
    • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance
    View case
  3. IL-0550 Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pending
    Atlassian2024
    Symptoms

    From 07:22 UTC on 22 February 2024, the bitbucket.org website and the Bitbucket Cloud APIs became slow, and Bitbucket Pipelines runs were badly delayed. Git requests over HTTPS and SSH kept working.

    High latencyConnection pool exhaustionIntermittent timeoutsThroughput degradation
    Root cause
    Capacity exhaustion
    Component
    DatabasePostgreSQL vacuum
    Contributing factor
    No alert on limit
    Remediation
    • Added resources to Bitbucket Pipelines to clear the backlog of jobs, which was fully cleared by 17:30 UTC
    Action items
    • Reconfigure the vacuuming threshold for database tables with high write activity
    • Adjust alert thresholds to catch this behaviour earlier
    • Tune autoscaling and load shedding for the Pipelines services and increase build runner capacity
    View case
Check your cluster for this

These 3 cases involve Database and PostgreSQL vacuum. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion3 cases

Component
Database3PostgreSQL vacuum1
Contributing factor
Circular dependency1Monitoring blind spot1No alert on limit1No capacity planning1
Organisation
Atlassian1Buttondown1Storj1
  1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed
    Buttondown2026
  2. IL-0557 Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hours
    Storj2025
  3. IL-0550 Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pending
    Atlassian2024