Cases2 cases

1 to 2 of 2

1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch up

   DeNA2026

   Symptoms

   On the event day of a global game, with close to 1000 rps per MySQL instance even after sharding and replication, a master instance stopped because of a host failure. MHA failed over automatically as usual and the service kept running, but one of the three replicas hit a replication error and could no longer follow the latest data.

   Replication lag

   Trigger

   Under heavy load on a game event day, a MySQL master instance stopped because of a host failure, and MHA started an automatic failover.

   Root cause

   Breaking change in an upgrade

   Component

   Database failover managerDatabase replicationDatabase

   Contributing factor

   Change with a delayed effect

   Remediation

   • Removed the replica that could not apply data from service

   Lessons

   MHA had not followed a mysqlbinlog change in MySQL 5.7.9, so under specific conditions bringing replicas up to date during failover failed. Running MySQL at large scale means meeting rare events like this, which only show when a master stops suddenly under load.

   View case
2. IL-0490 Records were missing from LootLocker's new MySQL database after a migration cutover and the service degraded for 3 hours

   LootLocker2026sev2

   Symptoms

   About an hour after LootLocker cut over from a managed database provider to its own MySQL deployment on 2026-04-28, data anomalies showed up in production. A small window of writes made shortly before the migration, spanning 2026-04-19 to 2026-04-20, had never reached the new database, and the service stayed degraded for about three hours after the cutover.

   Data loss

   Trigger

   On 2026-04-20 at about 16:20 BST, a routine credential rotation, made as part of an unrelated configuration fix, changed the credentials that the temporary replication process used for its database connection, and the process crashed.

   Root cause

   Certificate or credential expiry

   Component

   DatabaseDatabase replication

   Contributing factor

   Change with a delayed effectMonitoring blind spotUnverified assumptionNo safeguard on destructive change

   Detection

   Data anomalies were detected in production at about 11:30 BST, about an hour after the cutover.

   Remediation

   • Restored normal service behaviour first and deferred the root cause analysis
   • Requested a restore of the deleted source database from the previous provider
   • Replayed the affected records for impacted users from the restored source database

   Action items

   • Perform technical changes of comparable risk in a scheduled maintenance window instead of as live cutovers
   • Require a pre-cutover checklist that verifies replication or sync processes are live and caught up just before cutover
   • Require an explicit downstream impact review for credential rotations of service accounts used by long-running processes
   • Keep source systems until integrity in the target has been independently verified after cutover
   • Publish planned maintenance and migration notices in the in-product News feed as well as Discord, with clear impact statements and more advance notice

   Lessons

   The failure was a silent side effect of an unrelated change made eight days earlier: the replication process recovered and kept working, so there was no signal that data had been dropped, and a pre-cutover integrity check would have caught it. Deleting the source database before validating the new one made recovery much harder. Choosing a live migration to avoid downtime left no clean point to verify integrity, and the resulting degradation and two days of piecemeal recovery far exceeded what a short planned maintenance window would have cost. The announcement on Discord was not clear enough about the impact and came with too little notice.

   View case

By root cause2