IL-0356 Railway · 2026-09-30networking
Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed
Railway's distributed routing control plane tells its origin proxies where to send requests for each domain. It had recently been split out of a monolith so that it would keep running through a major cloud outage, and it runs in many places with rollout pipelines that run in parallel. A new authentication layer that lets customers put a login in front of their services needed a changed data type in the database table that stores routing data and a new routing service version that reads the new format. The schema update runs as a migration job in one system and the routing service is built and deployed by another. In the monolith a single pipeline had enforced the order; after the split, a configuration error let the migration and the deployment run in parallel, a race that shows most with larger migrations.
On 30 September 2026 the routing service build won the race. As it rolled out in every region it queried columns that did not exist yet, and every route lookup returned an error. The change also touched the domain fallback behaviour, so the origin proxies could not fall back to the last known route and returned 404 for every new connection to every domain, and private network DNS lookups failed too. At about 07:40 the CI system finished the schema update, the routing service's retried lookups returned 200, and domains routed normally again. Engineers found the cause at 07:44, confirmed full recovery at 07:47 without a rollback, posted a public incident at 07:48 and marked it resolved at 08:17. Railway states that no data was lost.
- Severity
- –
- Scope
- New connections to all domains hosted on Railway, custom and *.up.railway.app, returned HTTP 404; public access and some private networking lookups between services failed; existing connections and running workloads were not affected
- Users affected
- all customers
- Region
- all regions
- Time to detect
- 0m
- Time to resolve
- 5m
Symptoms
On 30 September 2026, between about 07:35 and 07:40 UTC, new connections to every domain hosted on Railway, both custom domains and *.up.railway.app domains, got HTTP 404 in all regions. Internal alerting paged the on-call team as soon as route lookups started to fail.
The 404 came from Railway's origin proxies. For each request, an origin proxy asks the routing service which workloads serve the domain. The routing service's lookups were failing, and the origin proxies answered a failed lookup the same way as a domain with no routes, with a 404, instead of using the last route they had seen for the domain. Private networking lookups between services also failed in the same window, because the same routing service resolves private network DNS.
Connections that were already open kept working, and the workloads behind the domains kept running; only new visits failed. Routing came back on its own about five minutes later, when the routing service's retried lookups started to succeed, and engineers confirmed full recovery at 07:47 without a rollback.
Root cause
Root cause Faulty deployment
- Trigger
- At 07:29 UTC an engineer merged a change to the routing control plane that contained a database schema update and a new version of the routing service. At about 07:35 the deployment system rolled the new routing service out to every region before the schema update had been applied.
Detection
Internal alerting engaged the on-call response team as soon as route lookups began to fail at about 07:35 UTC.
Remediation
- Let the CI system finish the schema update, after which the routing service's retried lookups succeeded and routing recovered without a rollback
Action items
- Force the migration rollout to run in serial order and unify the deployment pipeline so deploy order and blast radius are checked before merge
- Block the routing service's blue/green rollout until the database schema matching its version is in place
- Add shard based rollouts per region to the distributed control plane to isolate impact
Lessons
–
Sources
Related cases
IL-0324Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025same root cause · shared Database, CI system
IL-0552CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI · 2025same root cause · shared Database, CI system
IL-0136A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026same root cause · shared Database
IL-0354Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routesFly.io · 2026same root cause · shared Network control plane (SDN)
IL-0570Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minuteCrowdWorks · 2026same root cause · shared Database
IL-0234An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clustersIONOS Cloud · 2026same root cause · shared CI system
IL-0510MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hoursMIT Open Learning · 2026same root cause · shared Database
IL-0380A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeksInfluxData · 2026same root cause · shared Database
IL-0369Railway's backend API went down when a column migration waited on locks and replicas could not get database connectionsRailway · 2025same root cause · shared Database
IL-0547Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an indexBuildkite · 2025same root cause · shared Database
IL-0315Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connectionRailway · 2025same root cause · shared Database
IL-0327After a blue/green upgrade of CircleCI's workflows database every query hit disk, and workflow starts lagged or failedCircleCI · 2025same root cause · shared Database
IL-0563Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tablesLinode · 2025same root cause · shared Network control plane (SDN)
IL-0307Supabase's Postgres connection pooler Supavisor 2.0 dropped connections for about 80% of us-east-1 tenantsSupabase · 2025same root cause · shared Database
IL-0554A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutesSentry · 2024same root cause · shared Database
IL-0571STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutesSTORES · 2024same root cause · shared Database
IL-0293Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet lossGoogle Cloud · 2023same root cause · shared Database
IL-0530Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMsGoogle Cloud · 2023same root cause · shared Network control plane (SDN)
IL-0523TANF-app production deployments were blocked for two months when a search-index migration ran out of database disk spaceU.S. Department of Health and Human Services · 2023same root cause · shared Database
IL-0567Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not playDevsisters · 2023same root cause · shared Database
IL-0483Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran outReplicate · 2023same root cause · shared Database
IL-0371Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ranRender · 2023same root cause · shared Database
IL-0524An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hourOpen Collective · 2023same root cause · shared Database
IL-0328A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"CircleCI · 2021same root cause · shared Database
IL-0566Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenanceDevsisters · 2021same root cause · shared Database
IL-0467Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hoursKubernetes project · 2019same root cause · shared CI system
IL-0466Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own servicesKubernetes project · 2019same root cause · shared CI system
IL-0391GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lockGoCardless · 2015same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0356
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0356/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.