Cases2 cases

1 to 2 of 2

1. IL-0313 Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy pool

   Decagon2026

   Symptoms

   During the first production canary of Decagon's move from GCP's managed connection pooler to self-managed PgBouncer, a small number of voice requests stalled mid-conversation and timed out after exactly 300 seconds. Other requests failed much earlier because the SQLAlchemy connection pool was exhausted.

   Intermittent timeoutsConnection pool exhaustion

   Trigger

   The move from GCP's managed connection pooler to self-managed PgBouncer enabled a different TLS path between PgBouncer and Postgres, which supplied the last of three conditions needed to expose the bug.

   Root cause

   Software defect

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unrepresentative stagingUnverified assumption

   Detection

   Sentry errors ("SSL connection has been closed unexpectedly") and SQLAlchemy connection-pool timeouts during the production canary, followed by a comparison of application spans with Postgres query timings.

   Remediation

   • Shortened the application name parameter so the ParameterStatus packet stayed below PgBouncer's small-packet parsing threshold

   Action items

   • Fix PgBouncer upstream so it checks SSL_pending() before going back to wait on epoll()

   Lessons

   Decagon's lesson is that an infrastructure migration does not only change one's own systems but can also expose latent bugs elsewhere in the stack: the move to self-managed PgBouncer supplied the last condition for a long-standing edge case involving SQLAlchemy, PgBouncer, OpenSSL and the PostgreSQL wire protocol.

   View case
2. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory

   Railway2025

   Symptoms

   At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.

   High latencyIntermittent timeoutsDeploys failing

   Trigger

   After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionNo staged rolloutMonitoring blind spotUnreviewed defaults

   Detection

   Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.

   Remediation

   • Deployed a fix for the API failure caused by the schema change within 15 minutes
   • Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
   • Suspended all deployments, Pro included, for up to 20 minutes during peak instability
   • Scaled up the control plane database's CPU, memory, connection limits and IOPS
   • Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
   • Re-enabled deployments for all non-Pro users
   • Fixed the edge case in which private networks and their endpoints were missing from the dashboard

   Action items

   • Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
   • Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
   • Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
   • Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once

   Lessons

   Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.

   View case

By root cause2