IL-0570 CrowdWorks · 2026-09-07 (published)database
Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minute
The cause was a MySQL metadata lock (MDL). The dropped table had a foreign key to a table with very heavy traffic, and to keep foreign keys consistent MySQL requests an exclusive MDL on the parent table even when the child table is dropped. The migration waited to get that exclusive lock, and queries to the parent table queued up behind it, which the team considers the direct cause. The service's lock_wait_timeout was set to 50 seconds by default; the migration eventually failed on the timeout and the service recovered by itself. As prevention, CrowdWorks decided that such migrations set lock_wait_timeout = 3 for their session, so that a migration that cannot get the lock fails after 3 seconds. The team also considered the strong_migrations gem, which detects dangerous migrations, and a monkey patch that forces a short timeout on every migration, but DDL on specific high-traffic tables is rare and the extra dependency and maintenance were not judged worth it, so the rule is spread through documentation, AI review instructions and automatic pull request comments, agreed as an ADR (Architecture Decision Record).
- Severity
- –
- Scope
- crowdworks.jp was hard to reach for about a minute
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- 1m
Symptoms
Right after a deploy ran a migration with drop_table to remove a table that was no longer used, database queries started to pile up and crowdworks.jp was hard to reach for about a minute. The migration finally failed on a timeout and the service recovered on its own.
Symptom Lock contention or deadlockIntermittent connection failures
Root cause
Root cause Faulty deployment
- Trigger
- An engineer's deploy ran a migration that used drop_table to delete a table no longer in use.
Detection
–
Remediation
–
Action items
- In migrations that may take an exclusive MDL on a high-traffic table, through DDL on the table, adding or removing a foreign key to it, or creating or dropping a table with such a foreign key, run SET SESSION lock_wait_timeout = 3 first
- Write these migrations with def up and def down instead of def change, setting the timeout before the DDL in both
- Put only one such DDL in each migration
- Release migrations that contain such DDL on their own
- Run them during low-traffic hours
- Document the rules in db/migrate/README.md
- Point AGENTS.md at that README so Copilot and other AI agents review migrations against it
- Add the check to the automatic comment posted on pull requests that change db/migrate
Lessons
–
Sources
Related cases
IL-0369Railway's backend API went down when a column migration waited on locks and replicas could not get database connectionsRailway · 2025same root cause · shared Database, Schema migration
IL-0554A database migration used up Sentry US's Postgres connections, and the site and event ingestion were down for 22 minutesSentry · 2024same root cause · shared Database, Schema migration
IL-0523TANF-app production deployments were blocked for two months when a search-index migration ran out of database disk spaceU.S. Department of Health and Human Services · 2023same root cause · shared Database, Schema migration
IL-0524An index migration locked Open Collective's whole Transactions table and degraded the service for nearly an hourOpen Collective · 2023same root cause · shared Database, Schema migration
IL-0391GoCardless API requests timed out for about 15 seconds while a Postgres migration waited for an AccessExclusive lockGoCardless · 2015same root cause · shared Database, Schema migration
IL-0356Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026same root cause · shared Database
IL-0136A deployment that cleared an internal service's caches at peak traffic sent 20 times the normal load to the Azure OpenAI metadata backend in Sweden Central, and health checks that depended on its database restarted instances in a loop for almost six hoursMicrosoft Azure · 2026same root cause · shared Database
IL-0510MIT Learn's RDS PostgreSQL ran out of storage after a release and learn.mit.edu returned 503s and 504s for hoursMIT Open Learning · 2026same root cause · shared Database
IL-0380A compaction change in InfluxDB Cloud us-east-1 dropped data from compacted files, and queries missed data for weeksInfluxData · 2026same root cause · shared Database
IL-0547Buildkite annotation queries timed out and drove database CPU up across shards after a migration removed an indexBuildkite · 2025same root cause · shared Database
IL-0315Railway's backend API went down when an index build locked a billion-row Postgres table and used up every connectionRailway · 2025same root cause · shared Database
IL-0327After a blue/green upgrade of CircleCI's workflows database every query hit disk, and workflow starts lagged or failedCircleCI · 2025same root cause · shared Database
IL-0324Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database clusterWikimedia Foundation · 2025same root cause · shared Database
IL-0307Supabase's Postgres connection pooler Supavisor 2.0 dropped connections for about 80% of us-east-1 tenantsSupabase · 2025same root cause · shared Database
IL-0552CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UICircleCI · 2025same root cause · shared Database
IL-0571STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutesSTORES · 2024same root cause · shared Database
IL-0293Cloud Bigtable in us-central1 had higher latency and connection errors for about 32 hours from low-rate packet lossGoogle Cloud · 2023same root cause · shared Database
IL-0567Aurora MySQL behind Cookie Run: OvenBreak hit 100% CPU after a sales-ranking feature launch, and players could not playDevsisters · 2023same root cause · shared Database
IL-0483Replicate's website and API returned slow responses and HTTP 500s for two hours as PostgreSQL connections ran outReplicate · 2023same root cause · shared Database
IL-0371Half of new connections to Render Postgres databases with point-in-time recovery failed while their backup jobs ranRender · 2023same root cause · shared Database
IL-0328A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"CircleCI · 2021same root cause · shared Database
IL-0566Hot ranges after a Cookie Run: Kingdom guild update drove CockroachDB nodes to 100% CPU, forcing emergency maintenanceDevsisters · 2021same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0570
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0570/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.