Incident Library

Organisation

Hatena

Hatena · hatena

2 cases · 2017 to 2018

Root cause Configuration change1Operator error1
Component Database replication2Database failover manager2In-memory cache1Database1
Contributing factor Automation amplification2Overlapping failures2Unreviewed defaults1Unrepresentative staging1
Symptom Data loss1High latency1Intermittent timeouts1Monitoring outage1Network unreachable1Replication lag1

Cases

  1. IL-0430 Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degradedHatena · 2018

    Root cause Configuration change

  2. IL-0431 During Mackerel's move to AWS, PostgreSQL failed over to a replica that had stopped replicating, losing an hour of dataHatena · 2017

    Root cause Operator error