Cases2 cases

1 to 2 of 2

1. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw

   Wikimedia Foundation2023

   Symptoms

   Right after the primary datacenter was switched from EQIAD to CODFW at 14:00 UTC on 20 September 2023, on-wiki search on all Wikimedia wikis became delayed or unavailable for a subset of users. At 14:08 the alert "CirrusSearch codfw 95th percentile latency - more_like" fired and reached IRC and the Search Platform email list as "PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL".

   High latencyHigh error rateLoad imbalanceMisrouted requests

   Trigger

   At 14:00 UTC the primary datacenter was switched from EQIAD to CODFW. more_like queries that were expected to go to eqiad, the non-primary datacenter, went to codfw instead, apparently because RESTBase sent them as POST rather than GET.

   Root cause

   Software defect

   Component

   Elasticsearch or OpenSearchIn-memory cache

   Contributing factor

   Unverified assumptionDependence on a warm cacheNo runbook

   Detection

   Automated monitoring caught the problem, but not quickly enough to prevent user-facing impact: the Grafana alert "CirrusSearch codfw 95th percentile latency - more_like" went to IRC and the Search Platform email list at 14:08. No one was paged; SREs were already watching closely because of the datacenter switchover. At 14:14 Hashar pinged #wikimedia-search and engineers began troubleshooting.

   Remediation

   • Banned the overloaded node elastic2044 from the Elasticsearch cluster

   Action items

   • Work out which Elasticsearch steps are needed before a datacenter switchover
   • Update the Elasticsearch section of the datacenter switchover documentation
   • Review the suggested switchover improvements in T285347 and decide whether to implement them
   • Make the RESTBase /related/{article} endpoint send GET instead of POST so the CirrusSearch query cache stays warm in both datacenters
   • Review the shard balance of busy wikis, especially enwiki_content

   Lessons

   Monitoring caught the problem, but not quickly enough to prevent user-facing impact. The heightened attention during the switchover meant the impact was recognised immediately, and banning a single node was quick and easy. The mitigation was not guided by an existing runbook, and the shards on the overloaded host were not checked before it was banned.

   View case
2. IL-0217 Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outage

   Honeycomb2023

   Symptoms

   Hours after a routine switch between two clusters of Retriever, Honeycomb's storage and query engine, late on 24 July 2023, the internal SLO for Shepherd, the ingest service, began to burn slowly. The performance was only marginally slow and inconsistent, with no good explanation, so engineers treated it as minor and left it for the morning.

   High latencyDatabase overloadLock contention or deadlockService outage

   Trigger

   Late on 24 July Honeycomb switched between two Retriever clusters to avoid a subtle bug, and the new cluster stopped writing the last-written timestamps the schema cache depends on. On the morning of 25 July a feature flag flip meant to move those writes back stopped the remaining writes entirely.

   Root cause

   Software defect

   Component

   DatabaseIn-memory cache

   Contributing factor

   Dependence on a warm cacheIgnored early signalHarmful manual actionUnverified assumption

   Detection

   Hours after the cluster switch, the internal SLO for the ingest service Shepherd started burning slowly. Engineers left it for the morning, when a chart showed that Retriever's database calls updating the last-written records had stopped.

   Remediation

   • Flipped only the writes back to the other Retriever cluster with a feature flag, which did not fix the problem
   • Set up circuit breakers that denied all Shepherd traffic with a 5xx error to protect the database
   • Failed the hard-locked database over to a replica
   • Restarted all Retriever hosts so they could provide data for the cache
   • Set the last-written timestamp of every schema with traffic in the last day to now, so the schema store reloaded the cache
   • Removed the circuit breaker and restarted the query hosts that failed internal checks

   Action items

   • Finish the Retriever cluster migration and remove all code that could disable writes behind the cache
   • Explore ways to strengthen the cache further
   • Lower the contention put on the database during update storms from new schemas
   • Stabilise the performance costs of some operations in the system
   • Study instrumentation and experimentation that detect and handle such edge cases better

   Lessons

   Locally good decisions combined into the outage: the query engine wrote the timestamps because it already saw all the data, the shared cache helped scale ingest and untangle the main database, the cluster switch was done to avoid a bug, frequent deploys hid the feature flag bug, pausing deploys during the investigation kept hosts from restarting, and past near-misses made the team expect a slow failover. Feature flags, frequent deploys, pausing deploys during incidents and learning from near-misses all contributed, although they are among the practices Honeycomb trusts most. For a rare lockup of this kind a database failover is the fastest recovery. Several recent Honeycomb incidents started from trying to avoid a subtle bug.

   View case

By root cause1