Cases13 cases

1 to 5 of 13

1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed

   Buttondown2026

   Symptoms

   On 31 March 2026 Buttondown was down twice, for about seven and about thirteen minutes. New requests could not get a database connection and failed, although the database itself was healthy: queries were fast, CPU and memory were fine and replication lag was nominal.

   Connection pool exhaustionService outage

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningMonitoring blind spotCircular dependency

   Detection

   User reports and the team's own manual observation, not automated monitoring. The health check endpoint kept returning 200 for most requests, so no alert fired.

   Remediation

   • Killed active queries to free database connections in the first outage
   • Restarted the database to force-kill the running queries in the second outage

   Action items

   • Reserve a small number of database connections for administrative access
   • Monitor and alert on the database connection count as a percentage of the configured maximum
   • Make the health check endpoint run a lightweight query so connection exhaustion shows in the health checker and the load balancer
   • Add database connection limits to the quarterly capacity review
   • Document and script a fallback that force-recycles workers to release connections without needing a database connection

   Lessons

   Buttondown notes that the tool used to recover from the first outage needed the same resource that was exhausted, which locked the team out during the second outage, and that a health check which cannot detect connection exhaustion, the most common failure mode, gives little protection. A configuration value set long ago drifted out of line with traffic because nobody reviewed it.

   View case
2. IL-0317 Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429

   Clerk2026

   Symptoms

   At 8:11 AM PST (16:11 UTC) on 19 February 2026, Clerk's database performance dropped at once. Slow database queries made request handlers queue, and over 95% of traffic was answered with 429 without being handled; the small share of requests that reached the database returned 200, but extremely slowly.

   Service outageDatabase overloadHigh latency

   Trigger

   A routine automatic ANALYZE on one table sampled only NULL values in a column, and the Postgres planner switched a frequently run query to an inefficient plan.

   Root cause

   Query plan regression

   Component

   DatabaseQuery planner and table statistics

   Contributing factor

   Inefficient queryMonitoring blind spot

   Detection

   The incident was reported and escalated internally at 16:15 UTC, and the team started a group call.

   Remediation

   • Enabled a new failover that generates session tokens outside the core Session API
   • Re-ran ANALYZE manually on the affected table, which put the query back on its earlier plan
   • Disabled the failover once the database was back to healthy levels
   • Increased the statistics target on the table so the planner samples more rows
   • Refactored the query so the planner could plan it deterministically

   Action items

   • Add dedicated alerting for database query plan flips
   • Train the reliability team to enable the new session token failover and add automatic triggers for it
   • Audit all queries for others at risk of inefficient plans from unstable statistics
   • Name a dedicated communications lead during incidents
   • Post status page updates at a regular cadence even when nothing has changed
   • Make status page severity labels match the impact on customers
   • Cross-post incident updates to social channels

   Lessons

   Clerk says a query plan flip can cause sudden, severe degradation and needs to be detected directly instead of through downstream symptoms; relying on those symptoms led the team to blame a customer's traffic pattern. Customers also told Clerk that status page updates came too rarely, the first update came too late and the initial severity label did not match the impact.

   View case
3. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down

   Magic Pages2025

   Symptoms

   On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

   Service outage

   Trigger

   One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

   Root cause

   Dependency failure

   Component

   DatabasePersistent volumes and stateful sets

   Contributing factor

   Harmful manual actionSingle point of failureMonitoring blind spot

   Remediation

   • Restarted the storage server to fix PVC provisioning
   • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
   • Cleaned up the corrupted cluster metadata
   • Redeployed MySQL Router

   Action items

   • Monitor the health of the storage system more closely
   • Improve the failover procedures for MySQL Router
   • Investigate and fix the status page so it shows reliable information

   Lessons

   The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

   View case
4. IL-0373 Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four days

   Heroku2024

   Symptoms

   From 22:31 UTC on 27 July 2024, 33 Heroku Postgres clusters in the EU (Dublin) region could not serve requests and ran into memory problems, and some Essential-tier customers could not connect to their databases. Customers trying to reach an affected database received out of memory errors.

   Memory pressureService outage

   Trigger

   A third-party vendor deployed a change to a monitoring agent that runs on Heroku's Postgres database clusters.

   Root cause

   Software defect

   Component

   DatabaseTelemetry agent

   Contributing factor

   Monitoring blind spot

   Detection

   Heroku Support paged the technology team at 07:50 UTC on 28 July based on customer support tickets. The second occurrence also reached the technology team as an escalation from Heroku Support.

   Remediation

   • Allocated more resources and restarted the Postgres clusters, which fixed the first outage temporarily
   • Provisioned the previous version of the third-party monitoring software, which did not leak memory
   • The vendor applied an interim mitigation

   Action items

   • Add enhanced metrics collection for the third-party monitoring software
   • Evaluate the vendor's fixed release of the monitoring software and roll it out if acceptable

   View case
5. IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on them

   Microsoft Azure2024

   Symptoms

   From 21:40 UTC on 18 July 2024, virtual machines in the Azure Central US region lost access to their disks, and the services running on those VMs failed. Azure SQL Database and SQL Managed Instance showed unhealthy nodes and login failures, Cosmos DB requests failed, and service management operations failed across many Azure services.

   Storage disconnected or stale mountsService outageHigh error rateFailover failureStale endpoint routing

   Trigger

   Routine changes to the VM host fleet led to a new allow list of VM host address ranges being generated and published to all Storage scale units in Central US. Recent network configuration changes had made some of the infrastructure file servers that hold the source data unreachable from the server generating the list, so the published list was incomplete.

   Root cause

   Configuration change

   Component

   Cloud block storageDatabaseDatabase failover manager

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radiusMonitoring blind spot

   Detection

   Service monitoring detected the drop in VM availability at 22:06 UTC while the allow list update was being published. SQL monitoring detected unhealthy nodes and login failures within one minute of SQL becoming unavailable, and monitoring flagged the availability drop for Cosmos DB, SQL DB and SQL DB Managed Instance at 22:14 UTC.

   Remediation

   • Halted the allow list update workflow worldwide
   • Paused all deployments in the Central US region
   • Paused compute buildout deployments in all regions
   • Updated the configuration of all Storage scale units in Central US, restoring storage availability by 02:55 UTC
   • Started geo-failovers for SQL databases with automatic failover policies
   • Restarted SQL gateway nodes to refresh their caches and route connections to the right nodes
   • Forced completion of SQL failovers that had not completed
   • Failed over the Cosmos DB control plane and active-passive accounts with a write region in Central US
   • Cancelled customer-triggered graceful switch region operations that blocked Cosmos DB failovers
   • Removed locks on Cosmos DB service metadata to force-complete stuck failovers
   • Repaired impacted Cosmos DB partitions for single-region accounts
   • Repaired backend nodes and failed Cosmos DB accounts back to Central US, redoing failback where it failed

   Action items

   • Make the allow list generation workflow detect incomplete source information and halt
   • Alert on storage requests rejected by allow list checks
   • Deploy allow lists one Availability Zone and storage type at a time, over 24 hours
   • Add VM health checks and an automatic stop to the allow list deployment workflow
   • Reevaluate the policy for Microsoft-managed failover of SQL failover groups
   • Fix the Cosmos DB failback workflow that broke some MongoDB drivers for MongoDB API accounts
   • Fix Cosmos DB failback for accounts with private endpoints so it does not need redoing
   • Make the storage data-plane firewall reject invalid allow lists and keep the last-known-good state
   • Improve Azure NetApp Files monitors so impacting events are detected and classified in time
   • Monitor more Azure NetApp Files service metrics to correlate similar events faster
   • Make the SQL Service Fabric cluster location change notifications reliable under load
   • Make SQL geo-failover workflows more robust so they complete
   • Remove the issues that delayed 5% of Cosmos DB failovers
   • Migrate Azure DevOps customer metadata to the right geography
   • Decouple Azure NetApp Files regional reads and writes to keep this class of issue to one region
   • Evaluate caching to reduce Azure NetApp Files reliance on stored utilization data
   • Add automatic per-partition failover for Cosmos DB multi-region active-passive accounts
   • Build the Resilient Ephemeral OS disk improvement so VMs survive storage incidents better

   View case

By root cause8