Cases4 cases

1 to 4 of 4

1. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections

   Railway2025

   Symptoms

   At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

   Probe failuresLock contention or deadlockDatabase overloadService outage

   Trigger

   Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes

   Contributing factor

   Health check tied to a dependency

   Action items

   • Increase database connections and reconfigure PgBouncer to work within those limits
   • Fix the long-running query so it no longer causes row lock issues

   View case
2. IL-0554 A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutes

   Sentry2024

   Symptoms

   From 3:42 PM to 4:04 PM PT on 10 October 2024, the Sentry US site and event ingestion were unavailable, and the status page reported an issue with the Postgres database that made the site unavailable. The dashboard, the API and every US ingestion type, including errors, transactions, attachments, profiles, replays, spans, cron monitoring and custom metrics, were affected.

   Service outageDatabase overload

   Trigger

   A database migration ran certain database operations through the RunSQL command without lock and statement timeouts.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   No configuration testsNo runbook

   Remediation

   • Issued a fix, after which the dashboard and ingestion recovered

   Action items

   • Replace RunSQL commands with more robust solutions in the migration framework
   • Update the database framework to the latest version, which handles some operations RunSQL was used for
   • Write runbooks for recovering from lock or timeout issues during migrations
   • Improve alerting and observability so the team reacts sooner

   View case
3. IL-0524 An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hour

   Open Collective2023

   Symptoms

   At 09:20 UTC on 4 April 2023, seven minutes after a deployment was triggered, Open Collective's service degraded: a migration in the deployment had locked the entire Transactions table in the production database.

   Lock contention or deadlockDatabase overload

   Trigger

   A deployment ran a migration that dropped indexes on the Transactions table and re-created them with CONCURRENTLY, excluding soft-deleted rows.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   Missed in reviewUnrepresentative staging

   Detection

   The degradation of service was noticed at 09:20 UTC, seven minutes after the deployment was triggered.

   Remediation

   • Killed the migration, though the table remained locked
   • Put the frontend into maintenance mode to limit queries
   • Closed the other connections to the database
   • Cancelled the new index creation and restored the old index
   • Restored the frontend and confirmed the issue was resolved

   Action items

   • Add a script that simulates production load locally, using a tool like vegeta
   • Run this kind of migration during specific maintenance windows
   • Create the new index first and drop the old one afterwards in future migrations
   • Insist on a review from another engineer before merging such a pull request

   Lessons

   Dropping the old index before creating the new one was too optimistic under constant production queries, even with CONCURRENTLY, and an unreviewed pull request let it through. Tests locally and on staging did not show the problem because there were no tools to run migrations in stressed environments. Putting the frontend in maintenance mode limited the impact while showing users a clear message.

   View case
4. IL-0385 GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recovering

   GitHub2021

   Symptoms

   On 27 November 2021 at 20:40 UTC, a significant part of GitHub's MySQL read replicas entered a semaphore deadlock while a schema migration on a large MySQL table ran its final step, the rename that moves the updated table into place. Core GitHub services, including GitHub Actions, API Requests, Codespaces, Git Operations, Issues, GitHub Packages, GitHub Pages, Pull Requests and Webhooks, became degraded.

   Lock contention or deadlockDatabase overload

   Trigger

   The final rename step of a schema migration on a large MySQL table. Schema migrations are a common task at GitHub and often take weeks to complete.

   Root cause

   Software defect

   Component

   DatabaseSchema migration

   Contributing factor

   No headroomNo staged rollout

   Remediation

   • Promoted every healthy internal replica into the production path to add read capacity
   • Removed production traffic from the broken replicas until each had processed the table rename
   • Moved the recovered replicas back into production to restore capacity

   Action items

   • Continue functional partitioning so migrations can run in canary mode on a single shard
   • Update internal procedures to increase how much each cluster is over-provisioned
   • Pause schema migrations until it is known how to safeguard against this failure
   • Keep investigating the specific failure scenario
   • Test the migration tooling and classify ways to improve it for this kind of scenario

   View case

By root cause2