IL-0430 Hatena · 2018-09-26database

Mackerel's Redis stopped answering during a replication setup, keepalived failed over by mistake, and Mackerel degraded

Mackerel keeps monitoring data in Redis, and Redis memory use had been rising. On 26 September 2018 operators started building a Redis replica to scale Redis up. The replication setup took long enough that Redis did not answer for a time, keepalived, the clustering software used for Redis, judged the node to have failed, and an unintended failover followed at 10:51 JST. The application servers then could not reach Redis. Operators switched to the correct Redis at 10:55 and the application briefly recovered, but latency and timeouts came back, and rolling restarts of the application servers did not help.

Between 11:00 and 14:50 the team blocked requests from organizations that were posting inappropriate metrics, adjusted the application server timeouts, temporarily added capacity to Diamond (the TSDB), raised the number of Kinesis shards and Redis Cluster shards, and scaled out the API application servers. Mackerel was then put in maintenance mode, checked internally, and outside requests were let back in step by step. Recovery was declared at 15:20.

Hatena states that the failure started with the unintended failover, but it could not determine why recovery took so long and has stopped investigating. The review put forward one hypothesis: under a particular access pattern that continued for a while, the Scala application slowed down from waiting on synchronisation or lock contention in a thread pool or connection pool. An attempt to reproduce the situation in an isolated environment did not reproduce it.

Severity: –
Scope: The whole Mackerel service malfunctioned and alive monitoring stopped from 10:51 to 15:20 JST
Users affected: –
Region: –
Time to detect: –
Time to resolve: 4h 29m

Symptoms

On 26 September 2018 at 10:51 JST the whole Mackerel monitoring service stopped working properly and its alive monitoring stopped. The application servers could not connect to Redis, which stores Mackerel's monitoring data, and could not return correct responses.

At 10:55 operators switched the application back to the correct Redis and the application seemed to come back. It did not stay stable: application server latency got worse and requests timed out. Restarting the application servers one after another did not improve the symptoms.

The failure went on until 15:20 JST. Recovery was declared when application server responses were stable, the metrics that mackerel-agent resent had settled and the delay in writing data to the time series database (TSDB) was gone. A network problem and growing Redis memory use had already been present before the incident, and Hatena names both among the combined factors that kept the application unstable.

Symptom Monitoring outage High latency Intermittent timeouts

Root cause

Root cause Configuration change

Component In-memory cache Database replication Database failover manager

Contributing factor Automation amplification Overlapping failures

Trigger: Redis memory use had been growing, so operators set up replication in order to scale Redis up. Building the replication took a long time and Redis stopped answering for a while, and keepalived treated this as a node failure and started a failover nobody intended.

Detection

–

Remediation

Switched back to the correct Redis and briefly saw the application return
Restarted the application servers one after another, which did not improve the symptoms
Detected the organizations posting inappropriate metrics and blocked their requests
Adjusted the timeout interval of the application servers
Temporarily added capacity to Diamond, the TSDB
Increased the number of Kinesis shards and Redis Cluster shards
Scaled out the API application servers
Put Mackerel in maintenance mode, checked it internally and let outside requests back gradually

Action items

Raise the number of keepalived health checks for Redis so a brief load increase does not trigger a failover
Scale Redis up
Add more application servers
Change the application to store only the monitoring data it needs in Redis, reducing Redis memory use
Build a way to detect inappropriate API requests quickly and show them on a dashboard
Build a function to block organizations making inappropriate requests quickly
Keep improving the accuracy of detecting inappropriate requests and consider adding features such as an API limit
Strengthen monitoring of the application servers' internal processing so problems are handled before they become outages

Lessons

Hatena did not find why recovery took so long, but expects that keeping failover from reacting to short load spikes and giving the application more performance headroom would prevent an outage this long. It also aims to keep problems contained to a small part of the service when a failure does happen.

Sources

https://mackerel.io/ja/blog/entry/announcement/20181015 · blog · archived copy

Related cases

IL-0215 Twilio's billing Redis master lost all balances and went read-only, and auto-recharge charged customer cards repeatedlyTwilio · 2013
same root cause · shared Database replication, In-memory cache
IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checksAdGuard VPN · 2025
same root cause · shared In-memory cache
IL-0392 MariaDB replication of section x1 between eqiad and codfw broke on a missing heartbeat row during a cookbook runWikimedia Foundation · 2024
same root cause · shared Database replication
IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorumDigitalOcean · 2024
same root cause · shared Database replication
IL-0367 Azure Storage in Central US rejected VM disk I/O, taking down VMs and the SQL Database and Cosmos DB services on themMicrosoft Azure · 2024
same root cause · shared Database failover manager
IL-0331 An RDS MySQL Multi-AZ failover lost 5 seconds of committed data and broke replication on all five read replicasRazorpay · 2020
same root cause · shared Database replication
IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be readGoogle Cloud · 2017
same root cause · shared In-memory cache
IL-0325 DigitalOcean's control panel and API were down for 4 hours 56 minutes after the production primary database was deletedDigitalOcean · 2017
same root cause · shared Database replication

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Database replication, Database failover manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0430
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0430/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.