Incident Library
Applied filters FactorLimited debug accessOrganisationRailwayClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025
    Symptoms

    At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

    Service outageLock contention or deadlockDatabase overload
    Trigger

    A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

    Root cause
    Faulty deployment
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    No configuration testsLimited debug access
    Detection

    Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

    Remediation
    • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered
    Action items
    • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
    • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
    • Configure database user connection limits so administrative access stays possible during incidents
    View case
Check your cluster for this

This case involves Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
DatabaseDatabase proxy and connection pooler
Contributing factor
Limited debug accessNo configuration tests
Organisation
Railway
  1. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection
    Railway2025