Cases1 case

1 to 1 of 1

1. IL-0570 Dropping an unused MySQL table stalled queries on its busy parent table and crowdworks.jp was hard to reach for a minute

   CrowdWorks2026

   Symptoms

   Right after a deploy ran a migration with drop_table to remove a table that was no longer used, database queries started to pile up and crowdworks.jp was hard to reach for about a minute. The migration finally failed on a timeout and the service recovered on its own.

   Lock contention or deadlockIntermittent connection failures

   Trigger

   An engineer's deploy ran a migration that used drop_table to delete a table no longer in use.

   Root cause

   Faulty deployment

   Component

   DatabaseSchema migration

   Contributing factor

   –

   Action items

   • In migrations that may take an exclusive MDL on a high-traffic table, through DDL on the table, adding or removing a foreign key to it, or creating or dropping a table with such a foreign key, run SET SESSION lock_wait_timeout = 3 first
   • Write these migrations with def up and def down instead of def change, setting the timeout before the DDL in both
   • Put only one such DDL in each migration
   • Release migrations that contain such DDL on their own
   • Run them during low-traffic hours
   • Document the rules in db/migrate/README.md
   • Point AGENTS.md at that README so Copilot and other AI agents review migrations against it
   • Add the check to the automatic comment posted on pull requests that change db/migrate

   View case

By root cause1