Cases2 cases

1 to 2 of 2

1. IL-0434 Rolling restarts of a 30-node Elasticsearch 0.19 cluster after a host failure left 12 shards unassigned

   CyberAgent2016

   Symptoms

   After an OpenStack host server went down and took one Elasticsearch node with it, the health check that the API servers run against Elasticsearch returned false, although the Head plugin showed the cluster status as green. The check stayed false after the node was brought back and the cluster was green again.

   High latencyData loss

   Trigger

   An OpenStack host server went down, and one node of the Elasticsearch cluster went down with it.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unverified assumptionAutomation amplification

   Detection

   The health check that the API servers run against Elasticsearch returned false while the Head plugin showed the cluster as green.

   Remediation

   • Migrated the Elasticsearch node's virtual server to another host after the failed host recovered
   • Started the node's Elasticsearch process and let it rejoin the cluster
   • Restarted all nodes in order from node 1, pausing at about the twentieth when the API slowed
   • Restarted all nodes again from node 1, which ended the API delays but left 12 shards unassigned
   • Tried to move an unassigned shard with the allocate command of _cluster/reroute, which failed
   • For index A, copied node 22's shard files to nodes 3 and 10 and set primary to true in node 22's state file
   • For index B, deleted the broken shard files on node 27, copied node 28's files to node 27 and other nodes and marked node 28 primary
   • For index C, copied shards 2 and 4 from nodes 27 and 25 to two other nodes each and marked the copies as not primary

   Action items

   • Upgrade Elasticsearch and stop running such an old version

   Lessons

   Automatic shard reallocation should have been turned off before restarting nodes. After each node restart, the next node should have been restarted only after reallocation had completely finished, not just after the log file looked fine.

   View case
2. IL-0274 GitHub's code search Elasticsearch cluster hit near-100% CPU and rapid master elections; a restart found lost shards

   GitHub2013

   Symptoms

   On Thursday 24 January 2013 GitHub's exception tracking and monitoring systems detected a large spike in exceptions. Most of them were timeouts in code search queries and in the background jobs that update the code search indexes with data from new pushes, and the outage of the newly launched Code Search lasted into Friday 25 January.

   Intermittent timeoutsSustained high node CPUSplit brainData loss

   Trigger

   On 17 January GitHub upgraded the code search cluster from Elasticsearch 0.19.9 to 0.20.2 before launching Code Search publicly; the outage began on 24 January, after the public launch.

   Root cause

   Software defect

   Component

   Elasticsearch or OpenSearchLanguage runtime settings

   Contributing factor

   Unreviewed defaultsDeferred upgradesUntested changeIgnored early signalHarmful manual action

   Detection

   GitHub's exception tracking and monitoring systems detected a large spike in exceptions, mostly timeouts from code search queries and from the indexing background jobs.

   Remediation

   • Stopped the whole cluster and brought it back in maintenance mode, with shard allocation and rebalancing disabled
   • Brought in engineers from elasticsearch, the company, to review the cluster and its unallocatable shards
   • Upgraded Java and Elasticsearch and applied the recommended configuration through a Puppet topic branch and environment

   Action items

   • Provision a staging environment for the code search cluster to test infrastructure changes
   • Have the elasticsearch team audit the code search cluster configuration again in future

   Lessons

   The Elasticsearch upgrade was not tested enough on GitHub's infrastructure before it reached the code search cluster, partly because the cluster had no proper staging environment. For large clusters the elasticsearch team recommended equal minimum and maximum JVM heap through ES_HEAP_SIZE, at 50% of system RAM, starting recovery immediately instead of after recover_after_time, and configuring minimum_master_nodes so that paused nodes do not form their own clusters.

   View case

By root cause2