Cases4 cases

1 to 4 of 4

1. IL-0216 Wikimedia on-wiki search was slow or rejected after a datacenter switchover overloaded one Elasticsearch node in codfw

   Wikimedia Foundation2023

   Symptoms

   Right after the primary datacenter was switched from EQIAD to CODFW at 14:00 UTC on 20 September 2023, on-wiki search on all Wikimedia wikis became delayed or unavailable for a subset of users. At 14:08 the alert "CirrusSearch codfw 95th percentile latency - more_like" fired and reached IRC and the Search Platform email list as "PROBLEM alert - graphite1005/CirrusSearch codfw 95th percentile latency - more_like is CRITICAL".

   High latencyHigh error rateLoad imbalanceMisrouted requests

   Trigger

   At 14:00 UTC the primary datacenter was switched from EQIAD to CODFW. more_like queries that were expected to go to eqiad, the non-primary datacenter, went to codfw instead, apparently because RESTBase sent them as POST rather than GET.

   Root cause

   Software defect

   Component

   Elasticsearch or OpenSearchIn-memory cache

   Contributing factor

   Unverified assumptionDependence on a warm cacheNo runbook

   Detection

   Automated monitoring caught the problem, but not quickly enough to prevent user-facing impact: the Grafana alert "CirrusSearch codfw 95th percentile latency - more_like" went to IRC and the Search Platform email list at 14:08. No one was paged; SREs were already watching closely because of the datacenter switchover. At 14:14 Hashar pinged #wikimedia-search and engineers began troubleshooting.

   Remediation

   • Banned the overloaded node elastic2044 from the Elasticsearch cluster

   Action items

   • Work out which Elasticsearch steps are needed before a datacenter switchover
   • Update the Elasticsearch section of the datacenter switchover documentation
   • Review the suggested switchover improvements in T285347 and decide whether to implement them
   • Make the RESTBase /related/{article} endpoint send GET instead of POST so the CirrusSearch query cache stays warm in both datacenters
   • Review the shard balance of busy wikis, especially enwiki_content

   Lessons

   Monitoring caught the problem, but not quickly enough to prevent user-facing impact. The heightened attention during the switchover meant the impact was recognised immediately, and banning a single node was quick and easy. The mitigation was not guided by an existing runbook, and the shards on the overloaded host were not checked before it was banned.

   View case
2. IL-0241 A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one day

   Stripe2019

   Symptoms

   On 10 July 2019 the Stripe API failed a substantial majority of requests twice, from 16:35 to 17:02 UTC and from 21:14 to 22:47 UTC. In the first period automated monitoring detected within a minute that a database cluster had failed to elect a new primary.

   High error rateFailover failureWrites rejectedIntermittent timeoutsSustained high node CPU

   Trigger

   The primary node of a database cluster failed at 16:35 UTC while one shard had several stalled nodes. The second period was set off by rolling the election protocol back to the previous minor version at 20:42 UTC.

   Root cause

   Software defect

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Change with a delayed effectMonitoring blind spotExcessive blast radiusHarmful manual actionUnverified assumptionOverlapping failures

   Detection

   Automated monitoring detected the failed election within a minute, and the team was alerted at 16:36 UTC. The second period was announced at 21:14 UTC by automated alerts that some shards in the cluster were unavailable.

   Remediation

   • Restarted all nodes of the database cluster, which forced a successful primary election
   • Updated the production configuration and restarted the database cluster's nodes
   • Ramped traffic back up, putting services for user-initiated API requests first

   Action items

   • Alert when nodes stop reporting their replication lag
   • Alert when a shard enters a state that could trigger the election fault
   • Work with the database maintainers on a fix for the underlying fault
   • Add circuit breaking on failed operations to particular database clusters
   • Use fault isolation to contain the impact of a single failed shard
   • Limit the resources used by clients that keep retrying failed requests
   • Add procedures and tooling that make rapid configuration changes during incident response safer

   View case
3. IL-0279 One failed Redis Cluster replica left only about 1 in 21 calls to Grab's Apollo service succeeding for a minute

   Grab2019

   Symptoms

   On 15 February 2019 a slave node of the AWS ElastiCache Redis Cluster behind Apollo, Grab's driver-side state machine for transport bookings, failed and had to be replaced. For about one minute only roughly 1 in 21 calls to Apollo succeeded, a call failure rate above 95% against the Redis Cluster, and Grab rides were significantly down.

   High error rateStale endpoint routing

   Trigger

   A slave node of Apollo's Redis Cluster failed and needed to be replaced.

   Root cause

   Software defect

   Component

   In-memory cache

   Contributing factor

   Unverified assumption

   Lessons

   Having several replicas did not protect Apollo, because the client side, not the Redis Cluster server, caused the failure. Grab suggests three ways to avoid it: a separate Redis Cluster client used only for pipelining with RouteByLatency set to true, so pipelined reads go through the master nodes when latency is under 1ms, watching master CPU because they take more load; checking the error of each command in a pipeline instead of failing the whole call, so successful queries are not affected; and changing the Go-Redis library so that the pipeline path reloads the nodes' state when an error happens, as defaultProcessor already does.

   View case
4. IL-0263 GitHub was slow and erroring all day and down for 18 minutes as links between access and aggregation switches saturated

   GitHub2012

   Symptoms

   On Friday, 30 November 2012, GitHub had sporadic bursts of slow responses and intermittent errors for the whole day and was completely unavailable for 18 minutes. One fileserver pair was offline for a large part of the day, which affected a small percentage of repositories; this was a side effect of the network problems on the high-availability clustering between the fileserver nodes.

   High latencyHigh error rateService outage

   Trigger

   On the morning of 30 November GitHub moved the A switches in its server cabinets onto the new aggregation switches, the second half of a migration whose B-switch half had been done the day before.

   Root cause

   Software defect

   Component

   Network switchNIC, cable and switch port link

   Contributing factor

   Overlapping failuresUnverified assumptionHarmful manual action

   Detection

   Engineers saw the problems during the maintenance itself: connectivity to the B switch was lost as the last A switch was connected, and afterwards performance stayed low and the links between access and aggregation switches showed as completely saturated.

   Remediation

   • Fixed the bridge-loop problem and returned the affected B switch to service
   • Removed the partial link failure detection setting from the access switches
   • Escalated to the network vendor, which raised the case to its highest severity level
   • Restarted core processes on the aggregation switches in a Saturday 0600 Pacific maintenance window so they learned MAC addresses again

   Action items

   • Deploy the vendor's software update that fixes the MAC learning issue to the switches
   • Closely monitor aggregation-to-access capacity and keep a workaround process ready if the problem returns
   • Build a duplicate of the network stack, from the routers through the access switches, as a staging environment for testing such changes
   • Add automated network monitoring that alerts sooner on similar issues
   • Run scheduled incident response exercises with scenarios that counter tunnel vision
   • Contact the network vendor's support at the first sign of trouble in the network

   Lessons

   GitHub had designed the maintenance to have no impact on customers and clearly failed, so it plans to invest in a staging copy of its network to test such changes and catch bugs like this one. During the response the team fixated for a long time on a bridge loop, which kept it from considering other causes. GitHub had a very positive experience with the vendor's support staff, so it will contact that support team at the first sign of network trouble.

   View case

By root cause1