IL-0316 Railway · 2025-09-22database
Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory
The day began with a separate failure. During a code cleanup Railway removed an unused database column, a later code change unintentionally reverted part of that, and a core API that the dashboard relies on began to fail. From 11:05 UTC users reported that the project canvas would not load; the incident was declared at 11:14, a fix was deployed within 15 minutes and full restoration was reported at 11:45. The deployments that had queued meanwhile then started together, and from 11:48 the control plane showed high resource use and API latency. At 12:14 Railway linked the two problems and reopened the first incident.
Railway suspended deployments in phases to drain the queue and scaled up the control plane's database, but the problems continued. The cause was PgBouncer. Railway had used an older Bitnami image of PgBouncer and, after Bitnami deprecated its public images, had moved to an internal mirrored repository, which advanced PgBouncer by several versions. Under load the new version used abnormally high CPU and memory, and Railway identifies a breaking default configuration change in PgBouncer as the likely cause. Railway rolled PgBouncer back to the previous stable version instead of adjusting the configuration, saw a 100x performance increase with all metrics back to nominal, and had deployments restored for all tiers by 14:01 UTC. At 21:27 a rare edge case in which private networks and their endpoints no longer appeared on the dashboard was fixed retroactively.
- Severity
- –
- Scope
- Dashboard (project canvas stuck loading) and the deployment pipeline across Dashboard, GitHub webhooks and CLI; deployments suspended for non-Pro users and, for up to 20 minutes, for all users; running deployments and platform-level networking stayed online
- Users affected
- Users who opened the dashboard or started deployments during the incident
- Region
- –
- Time to detect
- –
- Time to resolve
- 2h 56m
Symptoms
At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.
Users saw an infinite loading state on their project canvas and delayed deployments, because the dashboard needs control plane APIs for data such as service domains and private network endpoints. Railway suspended deployments for Free, Trial and Hobby users at 11:48, and they saw a "Limited Access" message when trying to deploy; at 12:25 it suspended deployments for all tiers, Pro included, for up to 20 minutes. Some users also had issues with "Invalid database URL" or missing Private Networking.
The team first treated it as load and scaled up the control plane database's CPU and memory, tuned its connection limits and expanded its IOPS. Performance problems continued. Deployments came back for Pro users at 12:44 and non-Pro users at 12:50, but were withdrawn again for non-Pro users at 13:05 because the control plane stayed unstable. Further investigation found PgBouncer, the connection pooler in front of the control plane database, using far more CPU and memory than normal, which showed as increased latency and database connection timeouts.
Root cause
Root cause Breaking change in an upgrade
- Trigger
- After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.
Detection
Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.
Remediation
- Deployed a fix for the API failure caused by the schema change within 15 minutes
- Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
- Suspended all deployments, Pro included, for up to 20 minutes during peak instability
- Scaled up the control plane database's CPU, memory, connection limits and IOPS
- Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
- Re-enabled deployments for all non-Pro users
- Fixed the edge case in which private networks and their endpoints were missing from the dashboard
Action items
- Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
- Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
- Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
- Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once
Lessons
Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.
Sources
Related cases
IL-0304Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shardsIntercom · 2026same root cause · shared Database proxy and connection pooler, Database
IL-0498Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four daysClerk · 2025same root cause · shared Database proxy and connection pooler, Database
IL-0308Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slowRevenueCat · 2022same root cause · shared Database proxy and connection pooler, Database
IL-0572About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a dayMercari · 2026same root cause · shared Database
IL-0569After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026same root cause · shared Database
IL-0444Reader connections to Repro's Aurora MySQL cluster were cut with Lost connection errors every time ALTER TABLE ranRepro · 2024same root cause · shared Database
IL-0117A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024same root cause · shared Database
IL-0314Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restartingDaily · 2023same root cause · shared Database
IL-0294CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumersCircleCI · 2021same root cause · shared Database
IL-0372Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changedHeroku · 2021same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0316
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0316/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.