Cases5 cases

1 to 5 of 5

1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed

   Railway2026

   Symptoms

   On 30 September 2026, between about 07:35 and 07:40 UTC, new connections to every domain hosted on Railway, both custom domains and *.up.railway.app domains, got HTTP 404 in all regions. Internal alerting paged the on-call team as soon as route lookups started to fail.

   Service outageDNS resolution failures

   Trigger

   At 07:29 UTC an engineer merged a change to the routing control plane that contained a database schema update and a new version of the routing service. At about 07:35 the deployment system rolled the new routing service out to every region before the schema update had been applied.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)DatabaseCI system

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radius

   Detection

   Internal alerting engaged the on-call response team as soon as route lookups began to fail at about 07:35 UTC.

   Remediation

   • Let the CI system finish the schema update, after which the routing service's retried lookups succeeded and routing recovered without a rollback

   Action items

   • Force the migration rollout to run in serial order and unify the deployment pipeline so deploy order and blast radius are checked before merge
   • Block the routing service's blue/green rollout until the database schema matching its version is in place
   • Add shard based rollouts per region to the distributed control plane to isolate impact

   View case
2. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections

   Railway2025

   Symptoms

   At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

   Probe failuresLock contention or deadlockDatabase overloadService outage

   Trigger

   Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes

   Contributing factor

   Health check tied to a dependency

   Action items

   • Increase database connections and reconfigure PgBouncer to work within those limits
   • Fix the long-running query so it no longer causes row lock issues

   View case
3. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection

   Railway2025

   Symptoms

   At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

   Service outageLock contention or deadlockDatabase overload

   Trigger

   A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   No configuration testsLimited debug access

   Detection

   Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

   Remediation

   • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered

   Action items

   • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
   • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
   • Configure database user connection limits so administrative access stays possible during incidents

   View case
4. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memory

   Railway2025

   Symptoms

   At 11:48 UTC on 22 September 2025, minutes after Railway had resolved a dashboard outage, its control plane showed abnormal resource consumption and elevated API latency. A large backlog of deployments queued during the first outage had started all at once, and the control plane began returning elevated database errors.

   High latencyIntermittent timeoutsDeploys failing

   Trigger

   After a dashboard outage caused by a schema change was fixed, a large backlog of queued deployments started at the same time and put the control plane and its database, behind a newly upgraded PgBouncer, under heavy load.

   Root cause

   Breaking change in an upgrade

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   Unpinned versionNo staged rolloutMonitoring blind spotUnreviewed defaults

   Detection

   Users reported at 11:05 UTC that the project canvas failed to load; the control plane's abnormal resource consumption and elevated API latency were seen at 11:48 UTC and opened as a new incident.

   Remediation

   • Deployed a fix for the API failure caused by the schema change within 15 minutes
   • Suspended deployments for non-Pro users so Pro users could deploy and the queue could drain
   • Suspended all deployments, Pro included, for up to 20 minutes during peak instability
   • Scaled up the control plane database's CPU, memory, connection limits and IOPS
   • Rolled PgBouncer back to the previous stable version, which brought all metrics back to nominal
   • Re-enabled deployments for all non-Pro users
   • Fixed the edge case in which private networks and their endpoints were missing from the dashboard

   Action items

   • Re-engineer parts of the control plane to be more resilient, moving large parts of its storage layer away from Postgres
   • Add CI validation that keeps disruptive schema changes out of production and automated schema compatibility checks before backend deployments
   • Monitor and alert on PgBouncer node CPU, memory and connection pool utilization
   • Upgrade PgBouncer in phases, instance by instance, instead of everywhere at once

   Lessons

   Railway believes that without the breaking schema change reaching production the cascade would not have happened, that monitoring PgBouncer's own CPU, memory and pool metrics would have shown the performance regression earlier and shortened recovery, and that upgrading PgBouncer instances one at a time would have exposed the version-specific problem before it reached every instance.

   View case
5. IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutes

   Railway2024

   Symptoms

   Railway's edge servers on Google Cloud could not route inbound requests to applications in any region for about 10 minutes, on 17 December 2024 UTC. Private networking, outbound requests, Metal Hosts and Dedicated Instances kept working.

   Service outage

   Trigger

   A merged Terraform change emptied the array of a static IP block list, to unblock a customer whose IP had been blocked since before Railway moved to a dynamic block list and who had reported intermittent timeouts.

   Root cause

   Configuration change

   Component

   TerraformVPC route tables and firewall rules

   Contributing factor

   Unverified assumption

   Detection

   Automated alerting paged the current on-call engineer at 12:49 am UTC, two minutes after the change was merged.

   Remediation

   • Reverted the change to the IP allow-list rules
   • Applied the reverted change through the Terraform runner

   Action items

   • Automatically apply a page override for merged changes, for faster notification
   • Skip the Terraform plan phase during a revert
   • Remove the offending firewall rule safely
   • Audit all remaining legacy Google Cloud firewall rules for loose specificity
   • Move to its own metal infrastructure while keeping GCP and adding AWS, for resilience and fault isolation

   View case

By root cause3