Cases1 case

1 to 1 of 1

1. IL-0589 Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings table

   Kickstarter2013

   Symptoms

   On Thursday, March 7th, all of Kickstarter's MySQL replicas on Amazon RDS stopped applying updates, not just one replica that could be replaced. Most of Kickstarter's traffic is served from these replicas, so the site ran on stale copies of the database.

   Replication lag

   Trigger

   A later unsafe query on the backings table made MySQL switch to row-based replication for that query, and replaying the full row on the replicas collided with sequence values that already differed from the primary.

   Root cause

   Software defect

   Component

   DatabaseDatabase replication

   Contributing factor

   –

   Detection

   The team saw that the replicas had stopped updating, and SHOW SLAVE STATUS showed the duplicate-key replication error.

   Lessons

   Databases are intricate. In the Rails community they are sometimes treated as simple data stores hidden behind ActiveRecord and ActiveRelation, and Kickstarter's engineers concluded that it pays to learn how the database carries out what it is asked to do.

   View case

By root cause1