Cases3 cases

1 to 3 of 3

1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections

   Railway2025

   Symptoms

   At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

   Probe failuresLock contention or deadlockDatabase overloadService outage

   Trigger

   Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes

   Contributing factor

   Health check tied to a dependency

   Action items

   • Increase database connections and reconfigure PgBouncer to work within those limits
   • Fix the long-running query so it no longer causes row lock issues

   View case
2. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes

   Sentry2024

   Symptoms

   From 3:42 PM to 4:04 PM PT on 10 October 2024, the Sentry US site and event ingestion were unavailable, and the status page reported an issue with the Postgres database that made the site unavailable. The dashboard, the API and every US ingestion type, including errors, transactions, attachments, profiles, replays, spans, cron monitoring and custom metrics, were affected.

   Service outageDatabase overload

   Trigger

   A database migration ran certain database operations through the RunSQL command without lock and statement timeouts.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   No configuration testsNo runbook

   Remediation

   • Issued a fix, after which the dashboard and ingestion recovered

   Action items

   • Replace RunSQL commands with more robust solutions in the migration framework
   • Update the database framework to the latest version, which handles some operations RunSQL was used for
   • Write runbooks for recovering from lock or timeout issues during migrations
   • Improve alerting and observability so the team reacts sooner

   View case
3. IL-0391 GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lock

   GoCardless2015

   Symptoms

   Just after a planned Postgres schema migration started, GoCardless received alerts that API requests were timing out. The API was down for about 15 seconds, until the migration went through and the API came back.

   Lock contention or deadlockService outage

   Trigger

   A deploy with a schema migration that renamed the empty invoicing tables and dropped and recreated their foreign key constraints, which referenced some of the most heavily used tables.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   Unverified assumption

   Detection

   Alerts about API requests timing out began just after the schema migration started.

   Lessons

   Avoiding renames of tables in use, table rewrites under an exclusive lock and expensive work while holding one was not enough for GoCardless: a short DDL statement can still cause downtime by waiting in the lock queue behind a long-running query. GoCardless recommends removing long-running queries and transactions from the application and running analytics on an asynchronously updated replica, setting log_min_duration_statement and log_lock_waits to find such queries, setting lock_timeout in migration scripts so a deploy aborts rather than taking the application down, splitting schema changes so locks are held for shorter times, and keeping Postgres up to date. Developers with a console on the production database can cause the same downtime by leaving a transaction open.

   View case

By root cause1