Incident Library

IL-0274 GitHub · 2013-01-24database

GitHub's code search Elasticsearch cluster hit near-100% CPU and rapid master elections; a restart found lost shards

GitHub had moved all its search to Elasticsearch, with a dedicated code search cluster of 26 storage nodes and 8 client nodes; each storage node had 2TB of SSD storage. The cluster held roughly 17TB of code, about 34TB with one replica of every shard, and was 67% full. It had run on Java 6 and Elasticsearch 0.19.9 for several months while all code was backfilled into the index. On 17 January GitHub upgraded it to Elasticsearch 0.20.2 before the public launch. The 0.20.2 release had not been tested on GitHub's infrastructure or on any other cluster first, and there was no proper staging environment for the code search cluster. About two hours after the upgrade, during cluster recovery, 7 of the 510 shards were found missing both their primary and replica copies. The team put that down to high load during recovery and, after a healthy weekend, launched Code Search.

On 24 January several storage nodes went to nearly 100% CPU in the Elasticsearch Java process, nodes were rapidly elected to and removed from the master role, and code search queries and indexing jobs timed out. The team stopped the cluster and restarted it in maintenance mode. The 23 primaries that could not be allocated, with their replicas, had lost their data. GitHub brought in two engineers from elasticsearch, the company behind Elasticsearch, who found two bugs in Elasticsearch, released a fixed version within a few hours, and identified misconfigured or untuned settings. The Java 6 version GitHub ran dated from early 2009 and has several critical bugs affecting Elasticsearch and Lucene, plus problems with large memory allocations that can cause high load. The JVM's minimum and maximum memory differed, so the Java process blocked each time it grew its heap, which together with the old Java explains the pauses once public searches added load. minimum_master_nodes was not configured, so during long pauses subsets of nodes tried to form their own clusters. recover_after_time was set to 30 minutes, which delayed recovery.

GitHub rolled out upgrades of Java and Elasticsearch and the recommended configuration through a separate Puppet topic branch and environment. Each of the several restarts this needed made the cluster verify and recover its shards again, and loading 17TB of index from disk repeatedly was the slowest part of the outage; the audits added a few hours. With Elasticsearch 0.20.3, Java 7 and the new configuration, the erratic load and rapid master elections could no longer be reproduced. The outage affected only Code Search.

Severity
–
Scope
GitHub Code Search: search queries and index updates from new pushes timed out, and 23 primary shards with their replicas lost their data; no other GitHub component was affected
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On Thursday 24 January 2013 GitHub's exception tracking and monitoring systems detected a large spike in exceptions. Most of them were timeouts in code search queries and in the background jobs that update the code search indexes with data from new pushes, and the outage of the newly launched Code Search lasted into Friday 25 January.

On the Elasticsearch cluster dedicated to code search, a seemingly random subset of storage nodes carried massive load: most nodes used single-digit percentages of CPU, while several used nearly 100% of all their CPU cores. System load and IO load were ruled out; the only process behind the load was the Java process running Elasticsearch. While search and index timeouts continued, the logs showed several nodes being rapidly elected to the master role and removed from it again.

To stop the master role moving around the cluster, the team stopped the whole cluster and brought it back in "maintenance mode", which disables shard allocation and rebalancing. After the restart, some nodes could not rejoin the cluster at all, and some shards tried to allocate twice to the same node. The shards that could not be allocated, 23 primaries plus their replicas, had lost their data, and the cluster then spent a long time recovering the remaining shards.

Symptom Intermittent timeoutsSustained high node CPUSplit brainData loss

Root cause

Root cause Software defect

Component Elasticsearch or OpenSearchLanguage runtime settings
Contributing factor Unreviewed defaultsDeferred upgradesUntested changeIgnored early signalHarmful manual action
Trigger
On 17 January GitHub upgraded the code search cluster from Elasticsearch 0.19.9 to 0.20.2 before launching Code Search publicly; the outage began on 24 January, after the public launch.

Detection

GitHub's exception tracking and monitoring systems detected a large spike in exceptions, mostly timeouts from code search queries and from the indexing background jobs.

Remediation

Action items

Lessons

The Elasticsearch upgrade was not tested enough on GitHub's infrastructure before it reached the code search cluster, partly because the cluster had no proper staging environment. For large clusters the elasticsearch team recommended equal minimum and maximum JVM heap through ES_HEAP_SIZE, at 50% of system RAM, starting recovery immediately instead of after recover_after_time, and configuring minimum_master_nodes so that paused nodes do not form their own clusters.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Elasticsearch or OpenSearch, Language runtime settings among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.