Cases4 cases

1 to 4 of 4

1. IL-0313 Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy pool

   Decagon2026

   Symptoms

   During the first production canary of Decagon's move from GCP's managed connection pooler to self-managed PgBouncer, a small number of voice requests stalled mid-conversation and timed out after exactly 300 seconds. Other requests failed much earlier because the SQLAlchemy connection pool was exhausted.

   Intermittent timeoutsConnection pool exhaustion

   Trigger

   The move from GCP's managed connection pooler to self-managed PgBouncer enabled a different TLS path between PgBouncer and Postgres, which supplied the last of three conditions needed to expose the bug.

   Root cause

   Software defect

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unrepresentative stagingUnverified assumption

   Detection

   Sentry errors ("SSL connection has been closed unexpectedly") and SQLAlchemy connection-pool timeouts during the production canary, followed by a comparison of application spans with Postgres query timings.

   Remediation

   • Shortened the application name parameter so the ParameterStatus packet stayed below PgBouncer's small-packet parsing threshold

   Action items

   • Fix PgBouncer upstream so it checks SSL_pending() before going back to wait on epoll()

   Lessons

   Decagon's lesson is that an infrastructure migration does not only change one's own systems but can also expose latent bugs elsewhere in the stack: the move to self-managed PgBouncer supplied the last condition for a long-standing edge case involving SQLAlchemy, PgBouncer, OpenSSL and the PostgreSQL wire protocol.

   View case
2. IL-0329 A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outage

   incident.io2025

   Symptoms

   On 9 April 2025 from 14:16 UTC, incident.io's dashboard, mobile app, Slack app and API became intermittently slow and requests timed out. Monitoring alerted the team soon after a routine migration began on the production Postgres database, and the incident ended with the primary database unavailable for just under 2 minutes, from 14:25 to 14:27 UTC.

   Lock contention or deadlockIntermittent timeoutsHigh latencyService outage

   Trigger

   At 14:16 UTC a routine migration that created an empty table and added an index ran on the production database, where PGAudit had been re-enabled after the weekend upgrade to Postgres 17.

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   –

   Detection

   incident.io's monitoring alerted the team to the problem shortly after the migration started.

   Remediation

   • Restarted the primary database to force the stuck PGAudit process to end, taking just under 2 minutes of downtime
   • Disabled and then fully removed the PGAudit extension after service was restored

   Action items

   • Keep PGAudit removed until its safety can be guaranteed
   • Continue investigating the interaction between the migration and PGAudit
   • Hold an internal debrief of the incident
   • Improve processes and monitoring to detect, prevent and respond to similar situations

   View case
3. IL-0330 Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 seconds

   incident.io2023

   Symptoms

   Over two weeks in early 2023, incident.io's Go application intermittently timed out on requests that used its Postgres database. A customer reported the problem, Sentry showed a burst of context canceled errors, and the first episode was over within a minute.

   Connection pool exhaustionIntermittent timeouts

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   Monitoring blind spot

   Detection

   A customer report and a flurry of context canceled errors in Sentry; the on-call engineer opened an incident and checked the Grafana API dashboard. In the second episode an incident was opened automatically and the on-call engineer was pulled into it.

   Remediation

   • Moved the policy violations calculation to a materialised view
   • Added database indices for slow queries
   • Rewrote queries to filter on an indexed column already present in the join
   • Added a one second lock timeout to all transactions
   • Forwarded incoming Slack webhooks to PubSub and processed them asynchronously
   • Added a middleware that logs how long each operation holds a database connection
   • Removed the unnecessary transaction from most Slack modal submissions, keeping it only where needed

   Lessons

   The cause was not one slow operation but many short transactions that, added together, caused the problem. The investigation left incident.io with a way to filter logs and traces by how long each operation holds a database connection, and with real performance improvements. Separate databases for separate parts of the application would probably have made the cause easier to find, but would bring distributed transactions and less convenient development environments, a trade-off incident.io chose not to make for now.

   View case
4. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case

By root cause1