Incident Library

IL-0313 Decagon · 2026-08-05 (published)databasegcp

Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy pool

Decagon sets the PostgreSQL application name to the current request path, and whenever the value changes Postgres sends it back to the client in a ParameterStatus packet. The stall needed three conditions: an application name longer than 41 bytes, which makes the ParameterStatus packet larger than 64 bytes; a query response of a size that makes that packet cross PgBouncer's 4096-byte read buffer; and both parts of the split packet in the same TLS record between Postgres and PgBouncer. The first two had existed for some time, and the migration to self-managed PgBouncer added the third. Because GCP's managed pooler also uses PgBouncer and the data path stayed the same, Decagon had expected a simple migration, and staging seemed to confirm it.

PgBouncer's read loop waits for epoll() to report a readable socket, copies the data into its read buffer and parses PostgreSQL messages. With TLS, SSL_read() drains a whole TLS record from the kernel socket into OpenSSL's buffer and returns only as many plaintext bytes as requested, keeping the rest inside OpenSSL. In one reproduction Postgres sent a single 4,118-byte TLS record, PgBouncer read 4,096 bytes, and an 86-byte ParameterStatus packet crossed the boundary: PgBouncer saw 70 bytes, decided it needed more data and went back to epoll() without checking SSL_pending(), while the remaining 16 bytes sat in OpenSSL and nothing woke the connection. The query stayed blocked in cursor.execute() until Postgres killed the session after 300 seconds, so its connection stayed checked out of SQLAlchemy's process-level QueuePool. A few such connections shrank the pool, and unrelated requests sharing it failed with pool exhaustion.

Decagon shortened the application name so the ParameterStatus packet stayed below PgBouncer's small-packet parsing threshold, which removed one condition and let the migration continue. Upstream, Decagon merged a small fix to PgBouncer's event loop that checks SSL_pending() before sleeping and keeps reading when OpenSSL already holds plaintext.

Severity
–
Scope
During the production canary, a small number of voice requests stalled mid-conversation for 300 seconds, and unrelated requests sharing the same connection pool failed with pool exhaustion
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

During the first production canary of Decagon's move from GCP's managed connection pooler to self-managed PgBouncer, a small number of voice requests stalled mid-conversation and timed out after exactly 300 seconds. Other requests failed much earlier because the SQLAlchemy connection pool was exhausted.

Sentry showed a handful of "SSL connection has been closed unexpectedly" errors, and the pool failures read "sqlalchemy.exc.TimeoutError: QueuePool limit of size 2 overflow 20 reached, connection timed out, timeout 30.00". The SSL errors appeared only after PostgreSQL's idle-in-transaction session timeout ended the stalled sessions, so they looked like isolated connection failures with no link to the pool errors.

The pool error suggested the application had simply run out of database connections, but the other signals did not fit a capacity problem: Postgres query latency stayed at 1–3 milliseconds at p95, and PgBouncer's maxwait, the time clients wait for a server connection, stayed at zero. Comparing timelines showed that Postgres finished the affected queries in milliseconds while the application spans stayed open for exactly 300 seconds before returning the expected rows. The stalled queries kept their connections checked out of the QueuePool, and other requests timed out after 30 seconds waiting for those connections.

The stall was rare. Only two backend endpoints set an application name long enough to cause it, and only a small share of their query responses had the size that exposed the bug, so it never showed in staging.

Symptom Intermittent timeoutsConnection pool exhaustion

Root cause

Root cause Software defect

Component DatabaseDatabase proxy and connection pooler
Contributing factor Unrepresentative stagingUnverified assumption
Trigger
The move from GCP's managed connection pooler to self-managed PgBouncer enabled a different TLS path between PgBouncer and Postgres, which supplied the last of three conditions needed to expose the bug.

Detection

Sentry errors ("SSL connection has been closed unexpectedly") and SQLAlchemy connection-pool timeouts during the production canary, followed by a comparison of application spans with Postgres query timings.

Remediation

Action items

Lessons

Decagon's lesson is that an infrastructure migration does not only change one's own systems but can also expose latent bugs elsewhere in the stack: the move to self-managed PgBouncer supplied the last condition for a long-standing edge case involving SQLAlchemy, PgBouncer, OpenSSL and the PostgreSQL wire protocol.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database proxy and connection pooler among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.