Incident Library
Applied filters Root causeFaulty deploymentComponentSchema migrationOrganisationRailwayClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025
    Symptoms

    At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

    Probe failuresLock contention or deadlockDatabase overloadService outage
    Trigger

    Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes
    Contributing factor
    Health check tied to a dependency
    Action items
    • Increase database connections and reconfigure PgBouncer to work within those limits
    • Fix the long-running query so it no longer causes row lock issues
    View case
Check your cluster for this

This case involves Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
DatabaseDatabase proxy and connection poolerLiveness and readiness probesSchema migration
Contributing factor
Health check tied to a dependency
Organisation
Railway
  1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections
    Railway2025