Incident Library

IL-0315 Railway · 2025-10-28database

Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection

Railway's Dashboard, CLI and Public API depend on a backend that uses Postgres as its primary datastore. A routine change added a new column with an index to a table of about 1 billion records that nearly all API operations use. The index was created without Postgres' CONCURRENTLY option, so it took an exclusive lock on the whole table, and every query queued behind it. The API servers kept accepting requests and each new database connection blocked, and Railway's PgBouncer configuration allowed more connections than the Postgres limit, so all connection slots, including the administrative ones, were exhausted.

Engineers could not open an administrative connection to stop the index creation. While other recovery options were being explored, the migration finished on its own after about 30 minutes and released the table lock, the queued operations ran and the connection pools returned to normal. The root cause was identified at 18:56 UTC, gradual recovery was confirmed at 19:00 and full service at 19:15. Running deployments and platform-level features stayed online throughout.

Severity
–
Scope
Dashboard inaccessible, CLI operations that use the API failed, GitHub-based deployments delayed and all Public API operations disrupted; running deployments and platform-level features stayed online
Users affected
Users who opened the dashboard, used the CLI or started a GitHub-based deploy during the incident
Region
–
Time to detect
2m
Time to resolve
41m

Symptoms

At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

The change added a new column with an index to a table of about 1 billion records that nearly all API operations use. All queries against the database queued behind the index operation, and the API servers kept accepting requests whose database connections blocked on the locked table straight away. The connections that PgBouncer opened went past the Postgres connection limit and took every connection slot, the administrative ones included.

When engineers tried to connect and stop the index creation, Postgres refused with "FATAL: remaining connection slots are reserved for roles with privileges of the "pg_use_reserved_connections" role". Without an administrative connection, no command could be run to terminate the operation.

Symptom Service outageLock contention or deadlockDatabase overload

Root cause

Root cause Faulty deployment

Component DatabaseDatabase proxy and connection pooler
Contributing factor No configuration testsLimited debug access
Trigger
A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

Detection

Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database, Database proxy and connection pooler among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.