Cases2 cases

1 to 2 of 2

1. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory

   Railway2025

   Symptoms

   At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.

   High latencyIntermittent timeoutsDeploys failing

   Trigger

   After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionNo staged rolloutMonitoring blind spotUnreviewed defaults

   Detection

   Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.

   Remediation

   • Deployed a fix for the API failure caused by the schema change within 15 minutes
   • Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
   • Suspended all deployments, Pro included, for up to 20 minutes during peak instability
   • Scaled up the control plane database's CPU, memory, connection limits and IOPS
   • Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
   • Re-enabled deployments for all non-Pro users
   • Fixed the edge case in which private networks and their endpoints were missing from the dashboard

   Action items

   • Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
   • Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
   • Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
   • Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once

   Lessons

   Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.

   View case
2. IL-0314 Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restarting

   Daily2023

   Symptoms

   At 9:47 AM Eastern (14:47 UTC) on 7 February 2023, Daily's database alerts reported a performance problem under normal operational load: a high level of lock contention on a database that had been upgraded the weekend before and had run normally since Monday. Users could get websocket connection errors when joining calls, and REST API requests could time out or return 5xx errors.

   Lock contention or deadlockHigh error rateIntermittent timeouts

   Trigger

   Database maintenance the weekend before had upgraded the database to a MySQL point release with a low-level behaviour change, and on 7 February, under normal load, that change produced the deadlocks Daily saw on Aurora MySQL 2.11.0.

   Root cause

   Breaking change in an upgrade

   Component

   Database

   Contributing factor

   Unrepresentative stagingAutomation amplificationRetry stormChange with a delayed effect

   Detection

   Database alerts at 9:47 AM Eastern reported a performance issue with high lock contention on the newly upgraded database.

   Remediation

   • Disabled MySQL binlog replication to the old database and restarted the database to reduce load
   • The database restarted by itself at 10:47 AM, the SFUs dropped the stale meeting updates without restarting, and the platform recovered
   • Rate-limited the POST /rooms endpoint to the same limit as DELETE /rooms/:name
   • Updated an index on the affected table in production on the Saturday evening to work around the MySQL behaviour change

   Action items

   • Work with AWS so the Aurora MySQL 2.11.0 deadlock issue is documented, understood and fixed in a future release
   • Add more API request patterns to the testing workload used for infrastructure changes
   • Add production monitoring alarms on more fine-grained database metrics

   Lessons

   Daily tests infrastructure changes under production-like workloads, but its synthetic workload did not have the shape that triggers these deadlocks. A deadlock behaviour change in MySQL 5.7 that was already known turned out far more severe in production than either Daily or the AWS Aurora team expected.

   View case

By root cause1