IL-0314 Daily · 2023-02-07databaseaws
Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restarting
Daily upgraded the database that coordinates activity between its call servers over the weekend before 7 February 2023, and it ran normally on Monday. On Tuesday at 14:47 UTC alerts showed high lock contention, and within four minutes engineers found meeting-event queries from the SFUs stuck in deadlocks. API requests timed out, SFUs restarted, and each SFU shutdown and its replacement by autoscaling caused meeting moves that added more load. Disabling the leftover binlog replication to the old database and restarting the database at 10:13 Eastern helped only partly, because SFUs kept retrying the deadlocking queries. At 10:47, while engineers were writing a script to terminate stuck queries, the database restarted by itself. This restart took slightly longer than the one at 10:13, long enough for the SFUs to discard the stale meeting updates but not long enough to make them restart, and the platform returned to normal. The impact lasted almost exactly one hour, from 14:47 to 15:47 UTC, and the incident was moved to monitoring at 11:01.
Daily later proved that the deadlocks came from a low-level behaviour change in a MySQL point release that the weekend maintenance had brought in with Aurora MySQL 2.11.0. A more conservative approach to deadlocks was a known change in MySQL 5.7, on which Aurora MySQL 2 is based, but the severity surprised both Daily and the AWS Aurora team. As a precaution Daily rate-limited POST /rooms to the same limit as DELETE /rooms/:name, about 2 requests per second or 50 over 30 seconds, and later deployed a platform update after which the 429 rate limit responses stopped. The workaround for the behaviour change was to update an index on one affected table, which Daily planned and tested during the week and applied to the production database with no user impact on the Saturday evening.
- Severity
- –
- Scope
- API, Dashboard and Core Call Experience: some users in calls had meeting moves with video and audio dropping for a few seconds, a small number may have failed to join a room during a move, and some REST API requests returned 5xx errors
- Users affected
- Some users in Daily calls and some REST API clients
- Region
- –
- Time to detect
- 0m
- Time to resolve
- 1h
Symptoms
At 9:47 AM Eastern (14:47 UTC) on 7 February 2023, Daily's database alerts reported a performance problem under normal operational load: a high level of lock contention on a database that had been upgraded the weekend before and had run normally since Monday. Users could get websocket connection errors when joining calls, and REST API requests could time out or return 5xx errors.
By 9:51 engineers found a large volume of queries stuck in a deadlock. They were "meeting events" from the call servers (SFUs), recording when participants joined or left meetings. The deadlocks made webapp API requests time out and return 5xx errors, and the SFUs dropped their connections and restarted.
The problem fed on itself. An SFU shuts itself down when it cannot connect to the database, and autoscaling starts a new one to replace it. When an SFU shuts down, the webapp moves its call sessions to another SFU, a "meeting move" in which users see everyone else's video drop for a second or two and may need to restart recording or live streaming. With several SFUs shutting down at once, the extra meeting moves loaded the struggling database further.
At 10:13 the team suspected the MySQL binary log replication that still kept the old database up to date after the weekend migration, disabled it and restarted the database to reduce load. This helped, but many SFUs retried the queries that caused the deadlocks, so the problem persisted, and AWS support was contacted.
Symptom Lock contention or deadlockHigh error rateIntermittent timeouts
Root cause
Root cause Breaking change in an upgrade
- Trigger
- Database maintenance the weekend before had upgraded the database to a MySQL point release with a low-level behaviour change, and on 7 February, under normal load, that change produced the deadlocks Daily saw on Aurora MySQL 2.11.0.
Detection
Database alerts at 9:47 AM Eastern reported a performance issue with high lock contention on the newly upgraded database.
Remediation
- Disabled MySQL binlog replication to the old database and restarted the database to reduce load
- The database restarted by itself at 10:47 AM, the SFUs dropped the stale meeting updates without restarting, and the platform recovered
- Rate-limited the POST /rooms endpoint to the same limit as DELETE /rooms/:name
- Updated an index on the affected table in production on the Saturday evening to work around the MySQL behaviour change
Action items
- Work with AWS so the Aurora MySQL 2.11.0 deadlock issue is documented, understood and fixed in a future release
- Add more API request patterns to the testing workload used for infrastructure changes
- Add production monitoring alarms on more fine-grained database metrics
Lessons
Daily tests infrastructure changes under production-like workloads, but its synthetic workload did not have the shape that triggers these deadlocks. A deadlock behaviour change in MySQL 5.7 that was already known turned out far more severe in production than either Daily or the AWS Aurora team expected.
Sources
- https://status.daily.co/incidents/4r4b5hhz9w5v · official-status · archived copy
Related cases
IL-0572About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a dayMercari · 2026same root cause · shared Database
IL-0569After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026same root cause · shared Database
IL-0304Intercom's US region was fully down for 71 minutes when its Vitess VTGates dropped all routes to the database shardsIntercom · 2026same root cause · shared Database
IL-0316Railway's control plane slowed and deployments stopped after a PgBouncer version jump used abnormal CPU and memoryRailway · 2025same root cause · shared Database
IL-0498Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four daysClerk · 2025same root cause · shared Database
IL-0444Reader connections to Repro's Aurora MySQL cluster were cut with Lost connection errors every time ALTER TABLE ranRepro · 2024same root cause · shared Database
IL-0117A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start queryOpenAI · 2024same root cause · shared Database
IL-0308Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slowRevenueCat · 2022same root cause · shared Database
IL-0294CircleCI Docker and machine jobs stalled for nearly 12 hours after a VM cleanup queue lost all its RabbitMQ consumersCircleCI · 2021same root cause · shared Database
IL-0372Heroku apps could not connect to about 1% of hobby-tier Postgres databases after the databases' host names changedHeroku · 2021same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0314
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0314/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.