Incident Library

IL-0369 Railway · 2025-12-09database

Railway's backend API went down when a column migration waited on locks and replicas could not get database connections

Railway's backend runs as several replicas connected to a shared PostgreSQL database, with PgBouncer pooling the connections. A migration that added a nullable column to a heavily used table of about 1 billion rows stalled because a long-running query on the same table held locks. While the migration waited, connection attempts piled up and PgBouncer exceeded the database's connection limit, so the replicas could not obtain connections, failed their health checks and were taken out of rotation, which took the backend API down. When the long-running query finished and released its locks, the migration applied at once, the replicas reconnected and passed their health checks, and the backend was serving traffic again at 15:26 UTC, eight minutes after the lock conflicts began. The report's title and byline give 8 December 2025, while its timeline places the incident on 9 December 2025.

Severity
–
Scope
Railway backend API: dashboard access, CLI operations, GitHub-based deployment processing, login and API functionality; running deployments stayed online
Users affected
–
Region
–
Time to detect
–
Time to resolve
8m

Symptoms

At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

Each backend replica stays in rotation only while its periodic health check passes, and a replica that cannot get a database connection is marked unhealthy. Lock conflicts in the shared PostgreSQL database stalled a schema migration, connection attempts accumulated until PgBouncer went past the database's connection limit, and the replicas failed their health checks and were removed from rotation. All replicas recovered at 15:26 UTC, when the migration finished.

Symptom Probe failuresLock contention or deadlockDatabase overloadService outage

Root cause

Root cause Faulty deployment

Component DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes
Contributing factor Health check tied to a dependency
Trigger
Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

Detection

–

Remediation

–

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database proxy and connection pooler, Schema migration among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.