Cases2 cases

1 to 2 of 2

1. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory

   Railway2025

   Symptoms

   At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.

   High latencyIntermittent timeoutsDeploys failing

   Trigger

   After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionNo staged rolloutMonitoring blind spotUnreviewed defaults

   Detection

   Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.

   Remediation

   • Deployed a fix for the API failure caused by the schema change within 15 minutes
   • Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
   • Suspended all deployments, Pro included, for up to 20 minutes during peak instability
   • Scaled up the control plane database's CPU, memory, connection limits and IOPS
   • Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
   • Re-enabled deployments for all non-Pro users
   • Fixed the edge case in which private networks and their endpoints were missing from the dashboard

   Action items

   • Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
   • Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
   • Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
   • Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once

   Lessons

   Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.

   View case
2. IL-0582 Jobs piled up in Kanmu's PostgreSQL-based qg job queue on Aurora as dequeue slowed and dead tuples kept growing

   Kanmu2024

   Symptoms

   A Datadog alert fired because a large number of unprocessed jobs had piled up in Kanmu's job queue and were not being worked off. The jobs were the batch enqueued at about the same time every day, in the usual volume, but the workers took far longer than normal to dequeue and process them.

   Throughput degradationHigh latency

   Trigger

   A heavy ad hoc analysis query from the BI tool Redash ran for a long time on the writer instance, because the Redash data source pointed at the cluster endpoint.

   Root cause

   Operator error

   Component

   DatabasePostgreSQL vacuum

   Contributing factor

   Change with a delayed effectUnreviewed defaults

   Detection

   A Datadog alert on unprocessed jobs piling up in the job queue.

   Remediation

   • Changed the Redash data source to the reader endpoint
   • Ended the long-running Redash query with pg_terminate_backend
   • Ran VACUUM on que_jobs by hand again, which cleaned up the dead tuples

   Action items

   • Make Redash unable to connect to the writer instance even if the writer endpoint is set, by mapping its host name to a TEST-NET-1 address in the Redash container's /etc/hosts
   • Set statement_timeout on PostgreSQL users so that long queries stop automatically
   • Consider improving qg's dequeue algorithm or adopting a job queue such as river that uses SELECT FOR UPDATE SKIP LOCKED

   Lessons

   A long-running query on the same database instance, even one unrelated to que_jobs, can keep VACUUM from removing que_jobs dead tuples, so the table keeps growing and the job queue slows down. qg's design with advisory locks and a recursive CTE is sensitive to dead tuples.

   View case

By root cause2