Incident Library
Applied filters SymptomDNS resolution failuresFactorExcessive blast radiusOrganisationRailwayClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed
    Railway2026
    Symptoms

    On 30 September 2026, between about 07:35 and 07:40 UTC, new connections to every domain hosted on Railway, both custom domains and *.up.railway.app domains, got HTTP 404 in all regions. Internal alerting paged the on-call team as soon as route lookups started to fail.

    Service outageDNS resolution failures
    Trigger

    At 07:29 UTC an engineer merged a change to the routing control plane that contained a database schema update and a new version of the routing service. At about 07:35 the deployment system rolled the new routing service out to every region before the schema update had been applied.

    Root cause
    Faulty deployment
    Component
    Network control plane (SDN)DatabaseCI system
    Contributing factor
    Unsafe failure behaviourNo staged rolloutExcessive blast radius
    Detection

    Internal alerting engaged the on-call response team as soon as route lookups began to fail at about 07:35 UTC.

    Remediation
    • Let the CI system finish the schema update, after which the routing service's retried lookups succeeded and routing recovered without a rollback
    Action items
    • Force the migration rollout to run in serial order and unify the deployment pipeline so deploy order and blast radius are checked before merge
    • Block the routing service's blue/green rollout until the database schema matching its version is in place
    • Add shard based rollouts per region to the distributed control plane to isolate impact
    View case
Check your cluster for this

This case involves CI system and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
CI systemDatabaseNetwork control plane (SDN)
Contributing factor
Excessive blast radiusNo staged rolloutUnsafe failure behaviour
Organisation
Railway
  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed
    Railway2026