Cases1 case

1 to 1 of 1

1. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0

   Palark2021

   Symptoms

   After the last old Elasticsearch pod, elasticsearch-0, was deleted and came back on a new, empty persistent volume, `_cat/nodes` run inside that pod listed a single node: elasticsearch-0 itself, marked as master. The same command run inside elasticsearch-1 listed the other five nodes, elasticsearch-1 to elasticsearch-5, with elasticsearch-2 as master.

   Split brain

   Trigger

   Deleting the elasticsearch-0 pod and its PVC so that it restarted on an empty volume of the new StorageClass.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearchPersistent volumes and stateful setsHelm

   Contributing factor

   Unverified assumption

   Detection

   The engineer running the migration checked `_cat/nodes` after elasticsearch-0 restarted and saw it alone in a cluster of its own.

   Remediation

   • scaled the three deployments of workers writing to Elasticsearch down to zero
   • listed elasticsearch-0, elasticsearch-1 and elasticsearch-2 under initial_master_nodes in the elasticsearch ConfigMap and removed the cluster.initial_master_nodes environment variable from the StatefulSet
   • saved and deleted the elasticsearch discovery Service and deleted the PVC of elasticsearch-0 while the rolling update ran
   • set initial_master_nodes to elasticsearch-3, elasticsearch-4 and elasticsearch-5, created an empty PVC for elasticsearch-0 and deleted all six pods
   • recreated the discovery Service from the saved manifest, after which the six nodes formed one cluster with every index green

   Action items

   • Restore the original initial_master_nodes values in the ConfigMap and restart all pods again
   • Move all shards to nodes 0, 1 and 2 and scale the cluster down from six nodes to three
   • Commit all the manual changes to the repository

   Lessons

   A setting that only matters when a cluster first starts, such as Elasticsearch's cluster.initial_master_nodes, can sit in a Helm chart template rather than in the ConfigMap that was reviewed. When a pod named in it starts with an empty data directory, it can bootstrap a new cluster beside the existing one. The author's conclusion is that a migration plan like this one breaks on any surprise and needs a plan B ready. Before running it in a more important environment, rehearse it in a test environment configured like production, and either schedule downtime or move the load to another cluster: stop the writers, take a fresh backup, then move the data.

   View case

By root cause1