Cases1 case

1 to 1 of 1

1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch up

   DeNA2026

   Symptoms

   On the event day of a global game, with close to 1000 rps per MySQL instance even after sharding and replication, a master instance stopped because of a host failure. MHA failed over automatically as usual and the service kept running, but one of the three replicas hit a replication error and could no longer follow the latest data.

   Replication lag

   Trigger

   Under heavy load on a game event day, a MySQL master instance stopped because of a host failure, and MHA started an automatic failover.

   Root cause

   Breaking change in an upgrade

   Component

   Database failover managerDatabase replicationDatabase

   Contributing factor

   Change with a delayed effect

   Remediation

   • Removed the replica that could not apply data from service

   Lessons

   MHA had not followed a mysqlbinlog change in MySQL 5.7.9, so under specific conditions bringing replicas up to date during failover failed. Running MySQL at large scale means meeting rare events like this, which only show when a master stops suddenly under load.

   View case

By root cause1