Cases2 cases

1 to 2 of 2

1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed

   Railway2026

   Symptoms

   On 30 September 2026, between about 07:35 and 07:40 UTC, new connections to every domain hosted on Railway, both custom domains and *.up.railway.app domains, got HTTP 404 in all regions. Internal alerting paged the on-call team as soon as route lookups started to fail.

   Service outageDNS resolution failures

   Trigger

   At 07:29 UTC an engineer merged a change to the routing control plane that contained a database schema update and a new version of the routing service. At about 07:35 the deployment system rolled the new routing service out to every region before the schema update had been applied.

   Root cause

   Faulty deployment

   Component

   Network control plane (SDN)DatabaseCI system

   Contributing factor

   Unsafe failure behaviourNo staged rolloutExcessive blast radius

   Detection

   Internal alerting engaged the on-call response team as soon as route lookups began to fail at about 07:35 UTC.

   Remediation

   • Let the CI system finish the schema update, after which the routing service's retried lookups succeeded and routing recovered without a rollback

   Action items

   • Force the migration rollout to run in serial order and unify the deployment pipeline so deploy order and blast radius are checked before merge
   • Block the routing service's blue/green rollout until the database schema matching its version is in place
   • Add shard based rollouts per region to the distributed control plane to isolate impact

   View case
2. IL-0580 Deploys on a ConcourseCI pipeline failed intermittently with DNS lookup timeouts when pulling images from Amazon ECR

   Recruit2019

   Symptoms

   When an engineer at Recruit Technologies pressed the deploy button, Docker running in a ConcourseCI job failed to pull an image from Amazon ECR that docker build uses as cache. The error was: Error response from daemon: Get https://xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com/v2/: dial tcp: lookup xxxxxxxxxxxx.dkr.ecr.ap-northeast-1.amazonaws.com on 172.18.0.3:53: read udp 10.254.0.122:48372->172.18.0.3:53: i/o timeout

   DNS resolution failuresImage pull failures

   Trigger

   The Elastic Load Balancer behind the ECR endpoint returned more A records, as an ELB does when its units grow with request volume, which made the DNS responses larger.

   Root cause

   Software defect

   Component

   CI systemContainer registry and its credentialsClient-side DNS resolver and cache

   Contributing factor

   Unpatched known bug

   Detection

   The deploy failed with the DNS lookup timeout in the CI job when an engineer pressed the deploy button.

   Remediation

   • Changed the nameserver in resolv.conf on the CI worker to another DNS cache server as an interim fix

   Action items

   • Filed an issue and a pull request to Concourse to fix its DNS proxy forwarding uncompressed responses

   View case

By root cause2