Incident Library
Applied filters Root causeFaulty deploymentComponentDatabaseOrganisationRailwayClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed
    Railway2026
    Symptoms

    On 30 September 2026, between about 07:35 and 07:40 UTC, new connections to every domain hosted on Railway, both custom domains and *.up.railway.app domains, got HTTP 404 in all regions. Internal alerting paged the on-call team as soon as route lookups started to fail.

    Service outageDNS resolution failures
    Trigger

    At 07:29 UTC an engineer merged a change to the routing control plane that contained a database schema update and a new version of the routing service. At about 07:35 the deployment system rolled the new routing service out to every region before the schema update had been applied.

    Root cause
    Faulty deployment
    Component
    Network control plane (SDN)DatabaseCI system
    Contributing factor
    Unsafe failure behaviourNo staged rolloutExcessive blast radius
    Detection

    Internal alerting engaged the on-call response team as soon as route lookups began to fail at about 07:35 UTC.

    Remediation
    • Let the CI system finish the schema update, after which the routing service's retried lookups succeeded and routing recovered without a rollback
    Action items
    • Force the migration rollout to run in serial order and unify the deployment pipeline so deploy order and blast radius are checked before merge
    • Block the routing service's blue/green rollout until the database schema matching its version is in place
    • Add shard based rollouts per region to the distributed control plane to isolate impact
    View case
  2. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
    Symptoms

    At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

    Probe failuresLock contention or deadlockDatabase overloadService outage
    Trigger

    Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes
    Contributing factor
    Health check tied to a dependency
    Action items
    • Increase database connections and reconfigure PgBouncer to work within those limits
    • Fix the long-running query so it no longer causes row lock issues
    View case
  3. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
    Symptoms

    At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

    Service outageLock contention or deadlockDatabase overload
    Trigger

    A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    No configuration testsLimited debug access
    Detection

    Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

    Remediation
    • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered
    Action items
    • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
    • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
    • Configure database user connection limits so administrative access stays possible during incidents
    View case
Check your cluster for this

These 3 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment3 cases

Component
Database3Database proxy and connection pooler2CI system1Liveness and readiness probes1Network control plane (SDN)1Schema migration1
Contributing factor
Excessive blast radius1Limited debug access1No configuration tests1No staged rollout1Health check tied to a dependency1Unsafe failure behaviour1
Organisation
Railway3
  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed
    Railway2026
  2. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
  3. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025