Incident Library

Organisation

Daily

Daily · daily

1 case · 2023

Root cause Breaking change in an upgrade
Component Database
Contributing factor Automation amplificationChange with a delayed effectRetry stormUnrepresentative staging
Symptom High error rateIntermittent timeoutsLock contention or deadlock

Cases

  1. IL-0314 Daily's newly upgraded Aurora MySQL database deadlocked on meeting-event queries and its call servers kept restartingDaily · 2023

    Root cause Breaking change in an upgrade