IL-0569 DeNA · 2026-06-11 (published)database

After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch up

DeNA uses MHA, a MySQL high-availability tool that restores a correct replication topology with minimal downtime and data loss when a master fails. MySQL 5.7.9, released in 2015, changed mysqlbinlog to fix an error with GTID_MODE enabled (Bug #76887, Bug #20980932): when mysqlbinlog reads a Format_description_event (FDE), the metadata event at the start of a binlog, that the slave thread wrote into a relay log, it now prints only the comment # This Format_description_event appears in a relay log and was generated by the slave thread. and returns. A relay log starts with two FDEs, the slave's own and then the master's, and mysqlbinlog 5.7.8 still printed a BINLOG section for both.

During a failover MHA brings replicas up to date by joining log pieces in the order i1 (relay log received by a replica but not yet executed), i2 (relay log that one replica received and another did not) and X (binlog left on the old master that no replica received), and applies them through mysqlbinlog. The relay log always comes first, and MHA takes only the first FDE of the first file, which since 5.7.9 is no longer printed, so mysqlbinlog reaches a data update event before any usable FDE and fails. The error needs a replica on MySQL 5.7.9 or later and, during an MHA failover, either another replica that has read further into the old master's binlog (Read_Master_Log_Pos) or unexecuted relay log on this replica (Read_Master_Log_Pos different from Exec_Master_Log_Pos). That only happens when a master stops suddenly in the middle of transferring binlogs under real load, so the gap between MHA and the MySQL version stayed unnoticed for a long time and occurred only once in DeNA's environment.

When it did, the failover itself succeeded, but one of three replicas could not be brought up to date and then failed on later relay logs as well. The other instances were not affected, so removing this replica from service kept the impact on production small. The investigation happened some years before the article.

Severity

–

Scope

One of three replicas of the failed master could not apply data and was removed from service; the automatic failover kept the game service running

Users affected

–

Region

–

Time to detect

–

Time to resolve

–