Incident Library

Organisation

Railway

Railway · railway

5 cases · 2024 to 2026

Root cause Faulty deployment3Configuration change1Breaking change in an upgrade1
Component Database4Database proxy and connection pooler3CI system1Liveness and readiness probes1Network control plane (SDN)1Schema migration1Terraform1VPC route tables and firewall rules1
Contributing factor No staged rollout2Unverified assumption1Excessive blast radius1Unreviewed defaults1Limited debug access1Monitoring blind spot1No configuration tests1Health check tied to a dependency1Unpinned version1Unsafe failure behaviour1
Symptom Service outage4Database overload2Lock contention or deadlock2Deploys failing1DNS resolution failures1High latency1Intermittent timeouts1Probe failures1

Cases

  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026

    Root cause Faulty deployment

  2. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connectionsRailway · 2025

    Root cause Faulty deployment

  3. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connectionRailway · 2025

    Root cause Faulty deployment

  4. IL-0316 Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memoryRailway · 2025

    Root cause Breaking change in an upgrade

  5. IL-0355 Railway's Google Cloud edge stopped routing inbound requests to apps in all regions for about 10 minutesRailway · 2024

    Root cause Configuration change