Cases1 case

1 to 1 of 1

1. IL-0444 Reader connections to Repro's Aurora MySQL cluster were cut with Lost connection errors every time ALTER TABLE ran

   Repro2024

   Symptoms

   After Repro moved to Aurora MySQL 5.7, running an ALTER TABLE in production, even one that finished in under a second, made the processes reading that table fail with `Lost connection to MySQL server during query`. MySQL client connections were cut right after the ALTER TABLE ran, and many application errors followed. By 8 January 2021 this had happened three times in a row.

   Connection resetsHigh error rate

   Trigger

   Running ALTER TABLE on the Aurora MySQL writer while readers were running queries that used the same table.

   Root cause

   Breaking change in an upgrade

   Component

   Database

   Contributing factor

   Unrepresentative staging

   Remediation

   • Made the applications retry once after a connection error, with a monkey patch in the Rails applications

   View case

By root cause1