Cases3 cases

1 to 3 of 3

1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed

   Buttondown2026

   Symptoms

   On 31 March 2026 Buttondown was down twice, for about seven and about thirteen minutes. New requests could not get a database connection and failed, although the database itself was healthy: queries were fast, CPU and memory were fine and replication lag was nominal.

   Connection pool exhaustionService outage

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningMonitoring blind spotCircular dependency

   Detection

   User reports and the team's own manual observation, not automated monitoring. The health check endpoint kept returning 200 for most requests, so no alert fired.

   Remediation

   • Killed active queries to free database connections in the first outage
   • Restarted the database to force-kill the running queries in the second outage

   Action items

   • Reserve a small number of database connections for administrative access
   • Monitor and alert on the database connection count as a percentage of the configured maximum
   • Make the health check endpoint run a lightweight query so connection exhaustion shows in the health checker and the load balancer
   • Add database connection limits to the quarterly capacity review
   • Document and script a fallback that force-recycles workers to release connections without needing a database connection

   Lessons

   Buttondown notes that the tool used to recover from the first outage needed the same resource that was exhausted, which locked the team out during the second outage, and that a health check which cannot detect connection exhaustion, the most common failure mode, gives little protection. A configuration value set long ago drifted out of line with traffic because nobody reviewed it.

   View case
2. IL-0557 Database transaction locks piled up on Storj's US1 satellite, and uploads and downloads failed with timeouts for 2 hours

   Storj2025

   Symptoms

   From 21:04 UTC on 7 August 2025, database transaction locks on the Storj US1 satellite started to increase, and upload and download requests began to fail. The on-call team was paged at 21:14 UTC.

   Lock contention or deadlockIntermittent timeoutsConnection pool exhaustionHigh error rate

   Trigger

   An unusually high volume of concurrent uploads to the same objects reached the US1 satellite.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   –

   Detection

   The on-call team was paged at 21:14 UTC, ten minutes after database transaction locks began to rise.

   Remediation

   • Reset the state of connections periodically to keep contention errors from growing in a cascade

   Action items

   • Remove the sources of contention between database transactions with code changes
   • Add monitoring and alerts for increased levels of database locks
   • Look for further process and system improvements, including tools that reduce or remove the need for manual maintenance

   View case
3. IL-0550 Bitbucket Cloud's database slowed until connection pools saturated and requests timed out, leaving Pipelines pending

   Atlassian2024

   Symptoms

   From 07:22 UTC on 22 February 2024, the bitbucket.org website and the Bitbucket Cloud APIs became slow, and Bitbucket Pipelines runs were badly delayed. Git requests over HTTPS and SSH kept working.

   High latencyConnection pool exhaustionIntermittent timeoutsThroughput degradation

   Root cause

   Capacity exhaustion

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   No alert on limit

   Remediation

   • Added resources to Bitbucket Pipelines to clear the backlog of jobs, which was fully cleared by 17:30 UTC

   Action items

   • Reconfigure the vacuuming threshold for database tables with high write activity
   • Adjust alert thresholds to catch this behaviour earlier
   • Tune autoscaling and load shedding for the Pipelines services and increase build runner capacity

   View case

By root cause1