IL-0434 CyberAgent · 2016-06-10 (published)database

Rolling restarts of a 30-node Elasticsearch 0.19 cluster after a host failure left 12 shards unassigned

Ameba ran Elasticsearch 0.19.10 for a user-facing service on 30 OpenStack virtual servers with 24 cores and an 8GB heap each, holding 3 indices, about 300GB of data, 128 shards and 2 replicas. When an OpenStack host failed, one node went down and its shards were reallocated to the other nodes. After the host recovered, the node was migrated to another host in case the host failed again, its Elasticsearch process was started and it joined the cluster, which turned green. The API servers' health check against Elasticsearch still returned false. On another Elasticsearch environment a false health check had once been fixed by restarting all nodes, so the operators restarted all nodes here too, without turning off automatic shard reallocation and without waiting for reallocation to finish after each node. Reallocation ran over and over, the cluster became unstable and the API servers slowed down; a second full restart steadied the cluster but left 12 shards unassigned.

The shard files on disk explained why. For shard 53 of index A, directories existed on nodes 3, 10, 22 and 29 but only node 22 had files, and its state file said "primary" : false. For shard 51 of index B, nodes 27 and 28 both had data with different contents, and a lucene-core index check showed node 27's copy was broken and node 28's was healthy. For shards 2 and 4 of index C, files were left only on node 27 and node 25 respectively, with state files saying "primary" : true. With the Elasticsearch process stopped on the nodes involved, the operators copied the surviving files to the other nodes, deleted the broken copy on node 27, and edited the _state files so that exactly the surviving copy was marked primary true and the copies were false. After the processes were started, the Head plugin showed the shards back in the cluster.

Severity: –
Scope: 12 shards of the Elasticsearch cluster behind an Ameba user service stayed unassigned and part of the data was unavailable until the shard files were repaired by hand; the API servers were slow during the restarts
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

After an OpenStack host server went down and took one Elasticsearch node with it, the health check that the API servers run against Elasticsearch returned false, although the Head plugin showed the cluster status as green. The check stayed false after the node was brought back and the cluster was green again.

Operators then restarted every node one by one, starting from node 1 and checking the log files as they went. From about the twentieth node the API servers became slow, the restarts were paused, and the Head plugin could no longer show the cluster status: repeated shard reallocation had made the cluster unstable.

A second full restart from node 1 brought the Head plugin back, the API delays ended and the health check returned true. But 12 shards, including primary shards, did not come back into the cluster and stayed unassigned, so part of the data was lost to the service. Moving an unassigned shard with the allocate command of _cluster/reroute ended in an error.

Symptom High latency Data loss

Root cause

Root cause Operator error

Component Elasticsearch or OpenSearch

Contributing factor Unverified assumption Automation amplification

Trigger: An OpenStack host server went down, and one node of the Elasticsearch cluster went down with it.

Detection

The health check that the API servers run against Elasticsearch returned false while the Head plugin showed the cluster as green.

Remediation

Migrated the Elasticsearch node's virtual server to another host after the failed host recovered
Started the node's Elasticsearch process and let it rejoin the cluster
Restarted all nodes in order from node 1, pausing at about the twentieth when the API slowed
Restarted all nodes again from node 1, which ended the API delays but left 12 shards unassigned
Tried to move an unassigned shard with the allocate command of _cluster/reroute, which failed
For index A, copied node 22's shard files to nodes 3 and 10 and set primary to true in node 22's state file
For index B, deleted the broken shard files on node 27, copied node 28's files to node 27 and other nodes and marked node 28 primary
For index C, copied shards 2 and 4 from nodes 27 and 25 to two other nodes each and marked the copies as not primary

Action items

Upgrade Elasticsearch and stop running such an old version

Lessons

Automatic shard reallocation should have been turned off before restarting nodes. After each node restart, the next node should have been restarted only after reallocation had completely finished, not just after the log file looked fine.

Sources

https://ameblo.jp/principia-ca/entry-12168516931.html · blog · archived copy

Related cases

IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0Palark · 2021
same root cause · shared Elasticsearch or OpenSearch

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Elasticsearch or OpenSearch among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0434
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0434/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.