Cases1 case

1 to 1 of 1

1. IL-0314 Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restarting

   Daily2023

   Symptoms

   At 9:47 AM Eastern (14:47 UTC) on 7 February 2023, Daily's database alerts reported a performance problem under normal operational load: a high level of lock contention on a database that had been upgraded the weekend before and had run normally since Monday. Users could get websocket connection errors when joining calls, and REST API requests could time out or return 5xx errors.

   Lock contention or deadlockHigh error rateIntermittent timeouts

   Trigger

   Database maintenance the weekend before had upgraded the database to a MySQL point release with a low-level behaviour change, and on 7 February, under normal load, that change produced the deadlocks Daily saw on Aurora MySQL 2.11.0.

   Root cause

   Breaking change in an upgrade

   Component

   Database

   Contributing factor

   Unrepresentative stagingAutomation amplificationRetry stormChange with a delayed effect

   Detection

   Database alerts at 9:47 AM Eastern reported a performance issue with high lock contention on the newly upgraded database.

   Remediation

   • Disabled MySQL binlog replication to the old database and restarted the database to reduce load
   • The database restarted by itself at 10:47 AM, the SFUs dropped the stale meeting updates without restarting, and the platform recovered
   • Rate-limited the POST /rooms endpoint to the same limit as DELETE /rooms/:name
   • Updated an index on the affected table in production on the Saturday evening to work around the MySQL behaviour change

   Action items

   • Work with AWS so the Aurora MySQL 2.11.0 deadlock issue is documented, understood and fixed in a future release
   • Add more API request patterns to the testing workload used for infrastructure changes
   • Add production monitoring alarms on more fine-grained database metrics

   Lessons

   Daily tests infrastructure changes under production-like workloads, but its synthetic workload did not have the shape that triggers these deadlocks. A deadlock behaviour change in MySQL 5.7 that was already known turned out far more severe in production than either Daily or the AWS Aurora team expected.

   View case

By root cause1