Cases2 cases

1 to 2 of 2

1. IL-0313 Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy pool

   Decagon2026

   Symptoms

   During the first production canary of Decagon's move from GCP's managed connection pooler to self-managed PgBouncer, a small number of voice requests stalled mid-conversation and timed out after exactly 300 seconds. Other requests failed much earlier because the SQLAlchemy connection pool was exhausted.

   Intermittent timeoutsConnection pool exhaustion

   Trigger

   The move from GCP's managed connection pooler to self-managed PgBouncer enabled a different TLS path between PgBouncer and Postgres, which supplied the last of three conditions needed to expose the bug.

   Root cause

   Software defect

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unrepresentative stagingUnverified assumption

   Detection

   Sentry errors ("SSL connection has been closed unexpectedly") and SQLAlchemy connection-pool timeouts during the production canary, followed by a comparison of application spans with Postgres query timings.

   Remediation

   • Shortened the application name parameter so the ParameterStatus packet stayed below PgBouncer's small-packet parsing threshold

   Action items

   • Fix PgBouncer upstream so it checks SSL_pending() before going back to wait on epoll()

   Lessons

   Decagon's lesson is that an infrastructure migration does not only change one's own systems but can also expose latent bugs elsewhere in the stack: the move to self-managed PgBouncer supplied the last condition for a long-standing edge case involving SQLAlchemy, PgBouncer, OpenSSL and the PostgreSQL wire protocol.

   View case
2. IL-0330 Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 seconds

   incident.io2023

   Symptoms

   Over two weeks in early 2023, incident.io's Go application intermittently timed out on requests that used its Postgres database. A customer reported the problem, Sentry showed a burst of context canceled errors, and the first episode was over within a minute.

   Connection pool exhaustionIntermittent timeouts

   Root cause

   Software defect

   Component

   Database

   Contributing factor

   Monitoring blind spot

   Detection

   A customer report and a flurry of context canceled errors in Sentry; the on-call engineer opened an incident and checked the Grafana API dashboard. In the second episode an incident was opened automatically and the on-call engineer was pulled into it.

   Remediation

   • Moved the policy violations calculation to a materialised view
   • Added database indices for slow queries
   • Rewrote queries to filter on an indexed column already present in the join
   • Added a one second lock timeout to all transactions
   • Forwarded incoming Slack webhooks to PubSub and processed them asynchronously
   • Added a middleware that logs how long each operation holds a database connection
   • Removed the unnecessary transaction from most Slack modal submissions, keeping it only where needed

   Lessons

   The cause was not one slow operation but many short transactions that, added together, caused the problem. The investigation left incident.io with a way to filter logs and traces by how long each operation holds a database connection, and with real performance improvements. Separate databases for separate parts of the application would probably have made the cause easier to find, but would bring distributed transactions and less convenient development environments, a trade-off incident.io chose not to make for now.

   View case

By root cause1