Incident Library

Organisation

DeNA

DeNA · dena

2 cases · 2026

Root cause Operator error1Breaking change in an upgrade1
Component Database1Database replication1Database failover manager1Network switch1
Contributing factor Change during peak hours1Change with a delayed effect1Monitoring blind spot1No runbook1No staged rollout1
Symptom High latency1Intermittent connection failures1Replication lag1

Cases

  1. IL-0569 After an MHA master failover, one of three MySQL replicas failed with a malformed binlog error and could not catch upDeNA · 2026

    Root cause Breaking change in an upgrade

  2. IL-0568 A layer 2 loop after an access switch change broke network traffic across a whole DeNA site, with mass MACFLAP logsDeNA · 2026

    Root cause Operator error