Cases3 cases

1 to 3 of 3

1. IL-0579 Portfolio search latency on Amazon OpenSearch 1.3 spiked to about 1 second after migrating from Elasticsearch 6.8

   Classi2026

   Symptoms

   After Classi moved the search engine behind its Portfolio feature from Elasticsearch 6.8 to Amazon OpenSearch Service 1.3, the team saw in post-migration metrics that search responses were sometimes much slower. Most searches returned normally, but latency jumped at a certain frequency.

   High latency

   Trigger

   The migration from Elasticsearch 6.8 to OpenSearch 1.3, with indices that had no explicit refresh_interval.

   Root cause

   Breaking change in an upgrade

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed latency spikes in metrics checked after the migration and compared the maximum search latency before and after the migration in Datadog APM.

   Remediation

   • Set refresh_interval explicitly to 3s, which sharply reduced the latency spikes
   • Set refresh_interval explicitly to 1s in production

   Lessons

   The took value in OpenSearch's responses was already large, which could have narrowed the investigation to OpenSearch sooner. The slow log and the Profile API alone did not find the cause; asking when searches became slow, instead of where, led to the fix. The same setting can behave differently between versions, so defaults have to be checked as well.

   View case
2. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw

   Wikimedia Foundation2023

   Symptoms

   Right after the primary datacenter was switched from EQIAD to CODFW at 14:00 UTC on 20 September 2023, on-wiki search on all Wikimedia wikis became delayed or unavailable for a subset of users. At 14:08 the alert "CirrusSearch codfw 95th percentile latency - more_like" fired and reached IRC and the Search Platform email list as "PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL".

   High latencyHigh error rateLoad imbalanceMisrouted requests

   Trigger

   At 14:00 UTC the primary datacenter was switched from EQIAD to CODFW. more_like queries that were expected to go to eqiad, the non-primary datacenter, went to codfw instead, apparently because RESTBase sent them as POST rather than GET.

   Root cause

   Software defect

   Component

   Elasticsearch or OpenSearchIn-memory cache

   Contributing factor

   Unverified assumptionDependence on a warm cacheNo runbook

   Detection

   Automated monitoring caught the problem, but not quickly enough to prevent user-facing impact: the Grafana alert "CirrusSearch codfw 95th percentile latency - more_like" went to IRC and the Search Platform email list at 14:08. No one was paged; SREs were already watching closely because of the datacenter switchover. At 14:14 Hashar pinged #wikimedia-search and engineers began troubleshooting.

   Remediation

   • Banned the overloaded node elastic2044 from the Elasticsearch cluster

   Action items

   • Work out which Elasticsearch steps are needed before a datacenter switchover
   • Update the Elasticsearch section of the datacenter switchover documentation
   • Review the suggested switchover improvements in T285347 and decide whether to implement them
   • Make the RESTBase /related/{article} endpoint send GET instead of POST so the CirrusSearch query cache stays warm in both datacenters
   • Review the shard balance of busy wikis, especially enwiki_content

   Lessons

   Monitoring caught the problem, but not quickly enough to prevent user-facing impact. The heightened attention during the switchover meant the impact was recognised immediately, and banning a single node was quick and easy. The mitigation was not guided by an existing runbook, and the shards on the overloaded host were not checked before it was banned.

   View case
3. IL-0434 Rolling restarts of a 30-node Elasticsearch 0.19 cluster after a host failure left 12 shards unassigned

   CyberAgent2016

   Symptoms

   After an OpenStack host server went down and took one Elasticsearch node with it, the health check that the API servers run against Elasticsearch returned false, although the Head plugin showed the cluster status as green. The check stayed false after the node was brought back and the cluster was green again.

   High latencyData loss

   Trigger

   An OpenStack host server went down, and one node of the Elasticsearch cluster went down with it.

   Root cause

   Operator error

   Component

   Elasticsearch or OpenSearch

   Contributing factor

   Unverified assumptionAutomation amplification

   Detection

   The health check that the API servers run against Elasticsearch returned false while the Head plugin showed the cluster as green.

   Remediation

   • Migrated the Elasticsearch node's virtual server to another host after the failed host recovered
   • Started the node's Elasticsearch process and let it rejoin the cluster
   • Restarted all nodes in order from node 1, pausing at about the twentieth when the API slowed
   • Restarted all nodes again from node 1, which ended the API delays but left 12 shards unassigned
   • Tried to move an unassigned shard with the allocate command of _cluster/reroute, which failed
   • For index A, copied node 22's shard files to nodes 3 and 10 and set primary to true in node 22's state file
   • For index B, deleted the broken shard files on node 27, copied node 28's files to node 27 and other nodes and marked node 28 primary
   • For index C, copied shards 2 and 4 from nodes 27 and 25 to two other nodes each and marked the copies as not primary

   Action items

   • Upgrade Elasticsearch and stop running such an old version

   Lessons

   Automatic shard reallocation should have been turned off before restarting nodes. After each node restart, the next node should have been restarted only after reallocation had completely finished, not just after the log file looked fine.

   View case

By root cause3