Cases7 cases

4 to 7 of 7

1. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection

   Railway2025

   Symptoms

   At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

   Service outageLock contention or deadlockDatabase overload

   Trigger

   A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   No configuration testsLimited debug access

   Detection

   Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

   Remediation

   • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered

   Action items

   • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
   • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
   • Configure database user connection limits so administrative access stays possible during incidents

   View case
2. IL-0483 Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran out

   Replicate2023

   Symptoms

   From 05:45 UTC on 11 May 2023, for about two hours, many Replicate customers got slow responses or HTTP 500 errors from the API and the replicate.com website and had trouble running predictions. Monitoring paged an engineer at 05:46 UTC, a minute after the database connection pool became fully saturated.

   High error rateHigh latencyLock contention or deadlock

   Trigger

   A little after 05:00 UTC on 11 May, normal variations in traffic pushed the stacking INSERT queries past their tipping point.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Change with a delayed effect

   Detection

   Monitoring detected the saturated database connection pool almost immediately and paged an engineer at 05:46 UTC.

   Remediation

   • Switched off all asynchronous prediction update features with a feature flag at 07:31 UTC, after which the database recovered

   Action items

   • Keep the asynchronous prediction update feature disabled and redesign it so it no longer sends stacked INSERTs
   • Review lock contention and other database hotspots that could cause similar problems
   • Publish detailed reports after major incidents

   Lessons

   The system ran in a degraded state that looked successful until it tipped into a full failure. The same asynchronous update work that triggered the outage also partly protected customers who used webhooks, and the stacked INSERTs came directly from a latency improvement: removing 100-150ms brought two updates of one prediction to the database almost together.

   View case
3. IL-0524 An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hour

   Open Collective2023

   Symptoms

   At 09:20 UTC on 4 April 2023, seven minutes after a deployment was triggered, Open Collective's service degraded: a migration in the deployment had locked the entire Transactions table in the production database.

   Lock contention or deadlockDatabase overload

   Trigger

   A deployment ran a migration that dropped indexes on the Transactions table and re-created them with CONCURRENTLY, excluding soft-deleted rows.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   Missed in reviewUnrepresentative staging

   Detection

   The degradation of service was noticed at 09:20 UTC, seven minutes after the deployment was triggered.

   Remediation

   • Killed the migration, though the table remained locked
   • Put the frontend into maintenance mode to limit queries
   • Closed the other connections to the database
   • Cancelled the new index creation and restored the old index
   • Restored the frontend and confirmed the issue was resolved

   Action items

   • Add a script that simulates production load locally, using a tool like vegeta
   • Run this kind of migration during specific maintenance windows
   • Create the new index first and drop the old one afterwards in future migrations
   • Insist on a review from another engineer before merging such a pull request

   Lessons

   Dropping the old index before creating the new one was too optimistic under constant production queries, even with CONCURRENTLY, and an unreviewed pull request let it through. Tests locally and on staging did not show the problem because there were no tools to run migrations in stressed environments. Putting the frontend in maintenance mode limited the impact while showing users a clear message.

   View case
4. IL-0391 GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lock

   GoCardless2015

   Symptoms

   Just after a planned Postgres schema migration started, GoCardless received alerts that API requests were timing out. The API was down for about 15 seconds, until the migration went through and the API came back.

   Lock contention or deadlockService outage

   Trigger

   A deploy with a schema migration that renamed the empty invoicing tables and dropped and recreated their foreign key constraints, which referenced some of the most heavily used tables.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   Unverified assumption

   Detection

   Alerts about API requests timing out began just after the schema migration started.

   Lessons

   Avoiding renames of tables in use, table rewrites under an exclusive lock and expensive work while holding one was not enough for GoCardless: a short DDL statement can still cause downtime by waiting in the lock queue behind a long-running query. GoCardless recommends removing long-running queries and transactions from the application and running analytics on an asynchronously updated replica, setting log_min_duration_statement and log_lock_waits to find such queries, setting lock_timeout in migration scripts so a deploy aborts rather than taking the application down, splitting schema changes so locks are held for shorter times, and keeping Postgres up to date. Developers with a console on the production database can cause the same downtime by leaving a transaction open.

   View case

By root cause1