Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw

Symptoms

Right after the primary datacenter was switched from EQIAD to CODFW at 14:00 UTC on 20 September 2023, on-wiki search on all Wikimedia wikis became delayed or unavailable for a subset of users. At 14:08 the alert "CirrusSearch codfw 95th percentile latency - more_like" fired and reached IRC and the Search Platform email list as "PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL".

The PoolCounter rejected a share of search queries, up to 25% at the peak, for about half an hour. The rejections came from long response times, made worse by an unexpected fall in the cache hit rate from 71.6% to 55.5%. The more_like queries had been expected to go to eqiad, the non-primary datacenter, but were reaching codfw. At 14:26 Grafana showed extremely high load on one Elasticsearch node, elastic2044. After that node was banned from the cluster, latency began to return to normal, and at 14:35 the PoolCounter rejections were at zero.

Symptom High latencyHigh error rateLoad imbalanceMisrouted requests

Root cause

Trigger

At 14:00 UTC the primary datacenter was switched from EQIAD to CODFW. more_like queries that were expected to go to eqiad, the non-primary datacenter, went to codfw instead, apparently because RESTBase sent them as POST rather than GET.

Detection

Automated monitoring caught the problem, but not quickly enough to prevent user-facing impact: the Grafana alert "CirrusSearch codfw 95th percentile latency - more_like" went to IRC and the Search Platform email list at 14:08. No one was paged; SREs were already watching closely because of the datacenter switchover. At 14:14 Hashar pinged #wikimedia-search and engineers began troubleshooting.

Remediation

• Banned the overloaded node elastic2044 from the Elasticsearch cluster

Action items

• Work out which Elasticsearch steps are needed before a datacenter switchover
• Update the Elasticsearch section of the datacenter switchover documentation
• Review the suggested switchover improvements in T285347 and decide whether to implement them
• Make the RESTBase /related/{article} endpoint send GET instead of POST so the CirrusSearch query cache stays warm in both datacenters
• Review the shard balance of busy wikis, especially enwiki_content

Lessons

Monitoring caught the problem, but not quickly enough to prevent user-facing impact. The heightened attention during the switchover meant the impact was recognised immediately, and banning a single node was quick and easy. The mitigation was not guided by an existing runbook, and the shards on the overloaded host were not checked before it was banned.

Sources

• https://wikitech.wikimedia.org/wiki/Incidents/2023-09-20_Elasticsearch_unavailable · runbook · archived copy

References

• https://phabricator.wikimedia.org/T285347

Related cases

• IL-0376 A maintenance update set off needless full syncs between Upstash Redis replicas in eu-west-1, slowing the databasesUpstash · 2025

  same root cause · shared In-memory cache

• IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023

  same root cause · shared In-memory cache

• IL-0295 Pusher's Channels publish API on the mt1 cluster was largely unavailable for 11 minutes while a Redis master failed overPusher · 2022

  same root cause · shared In-memory cache

• IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021

  same root cause · shared In-memory cache

• IL-0041 fluent-bit silently stopped shipping application logs from one node to ElasticsearchPrometheusKube · 2020

  same root cause · shared Elasticsearch or OpenSearch

• IL-0279 One failed Redis Cluster replica left only about 1 in 21 calls to Grab's Apollo service succeeding for a minuteGrab · 2019

  same root cause · shared In-memory cache

• IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachableElastic · 2019

  same root cause · shared Elasticsearch or OpenSearch

• IL-0388 A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017

  same root cause · shared In-memory cache

• IL-0274 GitHub's code search Elasticsearch cluster hit near-100% CPU and rapid master elections; a restart found lost shardsGitHub · 2013

  same root cause · shared Elasticsearch or OpenSearch

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0216
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0216/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.