Cases2 cases

1 to 2 of 2

1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch up

   DeNA2026

   Symptoms

   On the event day of a global game, with close to 1000 rps per MySQL instance even after sharding and replication, a master instance stopped because of a host failure. MHA failed over automatically as usual and the service kept running, but one of the three replicas hit a replication error and could no longer follow the latest data.

   Replication lag

   Trigger

   Under heavy load on a game event day, a MySQL master instance stopped because of a host failure, and MHA started an automatic failover.

   Root cause

   Breaking change in an upgrade

   Component

   Database failover managerDatabase replicationDatabase

   Contributing factor

   Change with a delayed effect

   Remediation

   • Removed the replica that could not apply data from service

   Lessons

   MHA had not followed a mysqlbinlog change in MySQL 5.7.9, so under specific conditions bringing replicas up to date during failover failed. Running MySQL at large scale means meeting rare events like this, which only show when a master stops suddenly under load.

   View case
2. IL-0568 A layer 2 loop after an access switch change broke network traffic across a whole DeNA site, with mass MACFLAP logs

   DeNA2026

   Symptoms

   At about 20:00 on a weekday, one of DeNA's largest sites reported that terminals of a customer-facing system could not communicate and business was suffering. The network group's monitoring had raised no alert.

   Intermittent connection failuresHigh latency

   Trigger

   During a routine change on 15 Cisco Catalyst 2960L access switches at the site, which tidied descriptions, added VLAN definitions and added allowed VLANs on trunk ports, `no switchport trunk allowed vlan` was applied to GigabitEthernet0/15 of switch SW3, only one of the two physical ports in the LACP bundle between SW3 and SW2.

   Root cause

   Operator error

   Component

   Network switch

   Contributing factor

   Monitoring blind spotNo staged rolloutNo runbookChange during peak hours

   Detection

   The site reported the problem at about 20:00; no monitoring alert fired, because the affected port had no interface monitoring and LACP monitoring had been removed by an earlier monitoring change.

   Remediation

   • Rolled back the switch changes without knowing the cause; service recovered when the interface setting of one switch was restored
   • Found the faulty port and confirmed there was no other impact, completing the first response at about 22:00

   Action items

   • Fix the missing interface and LACP monitoring
   • Require a review for every monitoring change
   • Block merges to the alert rule repository without a reviewer's approval
   • Enable automatic AI review on pull requests
   • Announce every network change in advance, however minor
   • Agree the work date with the site's managers and avoid the site's event days
   • Prepare rollback procedures for high-impact maintenance and for work on several devices at once

   Lessons

   Any change can cause an outage, through a mistake, an unexpected event or a bug, so changes need to be scheduled with a maintenance window agreed in advance. A change on a downstream access switch can affect the whole site. Changing many devices at once made it slow both to find the faulty device and to roll back, and the parts that can be prevented by mechanisms rather than by people's care, such as reviews and monitoring, should be enforced that way.

   View case

By root cause2