Cases1 case

1 to 1 of 1

1. IL-0443 A Repro Aurora MySQL reader restarted unexpectedly and writer insert latency rose about sevenfold for a day or more

   Repro2019

   Symptoms

   A reader of Repro's Aurora MySQL cluster restarted for no known reason, and right after that the writer's insert latency rose to about seven times normal and writer performance stayed badly degraded for a long time. This happened more than once, twice within two weeks by 11 November 2019. While the writer was slow, the applications that process data collected by the Repro SDK fell behind, and the team could only watch until the writer recovered.

   High latencyReplication lagThroughput degradation

   Trigger

   Long-running queries on the reader; the author recalls that the cause was a query sent to the reader from Redash.

   Root cause

   Capacity exhaustion

   Component

   DatabaseDatabase replication

   Contributing factor

   Monitoring blind spot

   Remediation

   • Waited for writer performance to return, since the rollback segment release could not be sped up

   Action items

   • Add a monitor on the rollback segment, mysql.innodb.history_list_length in Datadog
   • Kill queries that run long enough to make the reader restart as soon as they are found
   • Set max_execution_time so queries are stopped after a fixed time

   Lessons

   Once the rollback segment has grown large, nothing speeds up its release across the cluster, so the way to avoid the latency is to keep transactions small and stop long-running queries before the rollback segment grows.

   View case

By root cause1