Cases1 case

1 to 1 of 1

1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed

   Buttondown2026

   Symptoms

   On 31 March 2026 Buttondown was down twice, for about seven and about thirteen minutes. New requests could not get a database connection and failed, although the database itself was healthy: queries were fast, CPU and memory were fine and replication lag was nominal.

   Connection pool exhaustionService outage

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningMonitoring blind spotCircular dependency

   Detection

   User reports and the team's own manual observation, not automated monitoring. The health check endpoint kept returning 200 for most requests, so no alert fired.

   Remediation

   • Killed active queries to free database connections in the first outage
   • Restarted the database to force-kill the running queries in the second outage

   Action items

   • Reserve a small number of database connections for administrative access
   • Monitor and alert on the database connection count as a percentage of the configured maximum
   • Make the health check endpoint run a lightweight query so connection exhaustion shows in the health checker and the load balancer
   • Add database connection limits to the quarterly capacity review
   • Document and script a fallback that force-recycles workers to release connections without needing a database connection

   Lessons

   Buttondown notes that the tool used to recover from the first outage needed the same resource that was exhausted, which locked the team out during the second outage, and that a health check which cannot detect connection exhaustion, the most common failure mode, gives little protection. A configuration value set long ago drifted out of line with traffic because nobody reviewed it.

   View case

By root cause1