Cases7 cases

1 to 5 of 7

1. IL-0570 Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minute

   CrowdWorks2026

   Symptoms

   Right after a deploy ran a migration with drop_table to remove a table that was no longer used, database queries started to pile up and crowdworks.jp was hard to reach for about a minute. The migration finally failed on a timeout and the service recovered on its own.

   Lock contention or deadlockIntermittent connection failures

   Trigger

   An engineer's deploy ran a migration that used drop_table to delete a table no longer in use.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   –

   Action items

   • In migrations that may take an exclusive MDL on a high-traffic table, through DDL on the table, adding or removing a foreign key to it, or creating or dropping a table with such a foreign key, run SET SESSION lock_wait_timeout = 3 first
   • Write these migrations with def up and def down instead of def change, setting the timeout before the DDL in both
   • Put only one such DDL in each migration
   • Release migrations that contain such DDL on their own
   • Run them during low-traffic hours
   • Document the rules in db/migrate/README.md
   • Point AGENTS.md at that README so Copilot and other AI agents review migrations against it
   • Add the check to the automatic comment posted on pull requests that change db/migrate

   View case
2. IL-0510 MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hours

   MIT Open Learning2026

   Symptoms

   On 24 March 2026 learn.mit.edu went down shortly after a regular release. An engineer reported the outage with a linked Sentry issue, and Sentry soon showed many 504 responses from the site.

   Service outageNode disk fullDatabase overloadLock contention or deadlock

   Trigger

   A regular release deployed a query pattern change to the learning resource APIs that put an aggregation inside the main query.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Inefficient queryUnrepresentative stagingHarmful manual action

   Detection

   An engineer reported the outage with a linked Sentry issue, and Sentry showed many 504 responses. The RDS FreeStorageSpace metric showed that the database had used up its storage.

   Remediation

   • Rolled back the release's query pattern change
   • Deleted the read replica so the database could flush its buffered WAL files
   • Terminated all active queries running longer than 10 minutes with pg_terminate_backend, after the backup finished and with guidance from AWS Support

   Action items

   • Explore enabling enhanced analytics on the production Learn database
   • Document rollback steps on the platform engineering documentation site
   • Make easy rollback part of the new release management framework
   • Alert on DiskQueueDepth for production databases without making the alert too noisy
   • Alert when API and site response times are more than 30% above normal
   • Run a gameday on an example production database outage
   • Adopt Rootly for incident management, with an automatic channel and an automatic incident transcript
   • Write an outage runbook: start a call, have the on-call engineer watch metrics until they return to baseline, and give one person the communications
   • Put the query that terminates long-running queries, and a version that finds them for selective termination, into a production Postgres runbook
   • Open an issue on redesigning the Learn Resource View Event model, which is only used for counts
   • Use anonymised production data generation tools to bring RC up to production data volume, on separate infrastructure for load testing
   • Change the view count to use a precomputed column

   Lessons

   The change was reviewed by senior engineers and showed no problems locally or on RC, but data scale and load in production are very different, and nothing in the system's health showed that adding a COUNT aggregation would make disk usage explode.

   View case
3. IL-0369 Railway's backend API went down when a column migration waited on locks and replicas could not get database connections

   Railway2025

   Symptoms

   At 15:18 UTC on 9 December 2025, Railway's backend replicas became unhealthy and stopped serving traffic. Dashboard access, CLI operations, logins, the API and the processing of GitHub-based deployments were disrupted, while running deployments stayed online.

   Probe failuresLock contention or deadlockDatabase overloadService outage

   Trigger

   Engineers merged a database schema change with a data migration at 15:03 UTC, and the migration began applying at 15:17 UTC. It added a nullable column to a heavily used table of about 1 billion rows.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection poolerSchema migrationLiveness and readiness probes

   Contributing factor

   Health check tied to a dependency

   Action items

   • Increase database connections and reconfigure PgBouncer to work within those limits
   • Fix the long-running query so it no longer causes row lock issues

   View case
4. IL-0315 Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connection

   Railway2025

   Symptoms

   At 18:36 UTC on 28 October 2025, two minutes after a database change went live in production, Railway's internal monitoring alerted and paged the engineering team. The dashboard could not be opened, CLI operations that use the API, such as railway up, failed, GitHub-based deployments were delayed because webhook processing failed, and all Railway Public API operations were disrupted.

   Service outageLock contention or deadlockDatabase overload

   Trigger

   A routine schema change to the backend's Postgres database, introduced at 18:23 UTC and live in production at 18:34 UTC, that added a new column with an index without the CONCURRENTLY option.

   Root cause

   Faulty deployment

   Component

   DatabaseDatabase proxy and connection pooler

   Contributing factor

   No configuration testsLimited debug access

   Detection

   Internal monitoring alerts paged the engineering team at 18:36 UTC, and the incident was declared at 18:41 UTC.

   Remediation

   • The index migration finished after about 30 minutes and released the table lock, and the queued operations and connection pools recovered

   Action items

   • Enforce CONCURRENTLY for all index creation in CI and block pull requests that do not use it
   • Adjust PgBouncer pool limits so they cannot overwhelm the Postgres instance's capacity
   • Configure database user connection limits so administrative access stays possible during incidents

   View case
5. IL-0483 Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran out

   Replicate2023

   Symptoms

   From 05:45 UTC on 11 May 2023, for about two hours, many Replicate customers got slow responses or HTTP 500 errors from the API and the replicate.com website and had trouble running predictions. Monitoring paged an engineer at 05:46 UTC, a minute after the database connection pool became fully saturated.

   High error rateHigh latencyLock contention or deadlock

   Trigger

   A little after 05:00 UTC on 11 May, normal variations in traffic pushed the stacking INSERT queries past their tipping point.

   Root cause

   Faulty deployment

   Component

   Database

   Contributing factor

   Change with a delayed effect

   Detection

   Monitoring detected the saturated database connection pool almost immediately and paged an engineer at 05:46 UTC.

   Remediation

   • Switched off all asynchronous prediction update features with a feature flag at 07:31 UTC, after which the database recovered

   Action items

   • Keep the asynchronous prediction update feature disabled and redesign it so it no longer sends stacked INSERTs
   • Review lock contention and other database hotspots that could cause similar problems
   • Publish detailed reports after major incidents

   Lessons

   The system ran in a degraded state that looked successful until it tipped into a full failure. The same asynchronous update work that triggered the outage also partly protected customers who used webhooks, and the stacked INSERTs came directly from a latency improvement: removing 100-150ms brought two updates of one prediction to the database almost together.

   View case

By root cause1