Cases2 cases

1 to 2 of 2

1. IL-0197 Moving a three-node Elasticsearch StatefulSet to new persistent volumes split the cluster in two, because the Helm chart hardcoded cluster.initial_master_nodes to elasticsearch-0

   Palark2021

   Symptoms

   After the last old Elasticsearch pod, elasticsearch-0, was deleted and came back on a new, empty persistent volume, `_cat/nodes` run inside that pod listed a single node: elasticsearch-0 itself, marked as master. The same command run inside elasticsearch-1 listed the other five nodes, elasticsearch-1 to elasticsearch-5, with elasticsearch-2 as master.

   Split brain

   Trigger

   Deleting the elasticsearch-0 pod and its PVC so that it restarted on an empty volume of the new StorageClass.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearchPersistent volumes and stateful setsHelm

   Contributing factor

   Unverified assumption

   Detection

   The engineer running the migration checked `_cat/nodes` after elasticsearch-0 restarted and saw it alone in a cluster of its own.

   Remediation

   • scaled the three deployments of workers writing to Elasticsearch down to zero
   • listed elasticsearch-0, elasticsearch-1 and elasticsearch-2 under initial_master_nodes in the elasticsearch ConfigMap and removed the cluster.initial_master_nodes environment variable from the StatefulSet
   • saved and deleted the elasticsearch discovery Service and deleted the PVC of elasticsearch-0 while the rolling update ran
   • set initial_master_nodes to elasticsearch-3, elasticsearch-4 and elasticsearch-5, created an empty PVC for elasticsearch-0 and deleted all six pods
   • recreated the discovery Service from the saved manifest, after which the six nodes formed one cluster with every index green

   Action items

   • Restore the original initial_master_nodes values in the ConfigMap and restart all pods again
   • Move all shards to nodes 0, 1 and 2 and scale the cluster down from six nodes to three
   • Commit all the manual changes to the repository

   Lessons

   A setting that only matters when a cluster first starts, such as Elasticsearch's cluster.initial_master_nodes, can sit in a Helm chart template rather than in the ConfigMap that was reviewed. When a pod named in it starts with an empty data directory, it can bootstrap a new cluster beside the existing one. The author's conclusion is that a migration plan like this one breaks on any surprise and needs a plan B ready. Before running it in a more important environment, rehearse it in a test environment configured like production, and either schedule downtime or move the load to another cluster: stop the writers, take a fresh backup, then move the data.

   View case
2. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured

   Devsisters2021

   Symptoms

   During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

   Node NotReadyData lossService outage

   Trigger

   A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

   Root cause

   Cloud provider outage

   Component

   Application databaseHelm

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A Slack alert reported that a database node had gone down.

   Remediation

   • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
   • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
   • restored the last data range with the storage-layer recovery method from the previous incident

   Action items

   • Configure CockroachDB Locality so data is spread evenly across availability zones
   • Deploy with one chart and a separate values file per availability zone to set Locality
   • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes

   Lessons

   A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

   View case

By root cause2