Incident Library

IL-0279 Grab · 2019-02-15databaseaws

One failed Redis Cluster replica left only about 1 in 21 calls to Grab's Apollo service succeeding for a minute

Apollo stores the availability of individual drivers in an AWS ElastiCache Redis Cluster of 3 shards, each with 2 slaves. Apollo uses the Go-Redis client: writes go to the master nodes, and for reads the ReadOnly flag is on and RouteByLatency is off, which in Grab's gredis3 library turns on ReadOnlyFromSlaves so that reads go to slave nodes and spare the masters' CPU. When the slave failed, the Go-Redis pipeline code kept mapping read commands to the slave nodes from its cached cluster state, which the client refreshes only once a minute, so for up to a minute it kept sending reads to the unreachable slave and all commands assigned to it failed. Grab's Redis wrapper returned an error for the whole pipeline when any single command failed, and Apollo's code treated any error as a failure, which produced the high failure rate. Writes kept working because the client watches the master nodes and knows their state at once. The Redis Cluster recovered on its own after about one minute.

Severity
–
Scope
Apollo, Grab's driver-side state machine on the critical path of transport bookings: about 1 in 21 calls succeeded for one minute and Grab rides were significantly down
Users affected
–
Region
–
Time to detect
–
Time to resolve
1m

Symptoms

On 15 February 2019 a slave node of the AWS ElastiCache Redis Cluster behind Apollo, Grab's driver-side state machine for transport bookings, failed and had to be replaced. For about one minute only roughly 1 in 21 calls to Apollo succeeded, a call failure rate above 95% against the Redis Cluster, and Grab rides were significantly down.

The errors came from pipeline GET calls: when Apollo sent a pipeline of HMGET commands to the Redis Cluster, the pipeline returned errors. The cluster has 3 shards with 2 slaves each, so a single slave failure should have left it fully available, and the Redis Cluster server broadcast the unreachable node to the other nodes at once. Writes, which go only to master nodes, were not affected and no new data was lost.

Grab first looked at the Redis Cluster server and, after reproducing node failures on a local cluster, concluded that the server handled the failure correctly and was not the cause of the failure rate. The investigation then moved to the client side, the Go-Redis library and Apollo's queries, and to why recovery took almost a minute when the cluster had broadcast the change immediately.

Symptom High error rateStale endpoint routing

Root cause

Root cause Software defect

Component In-memory cache
Contributing factor Unverified assumption
Trigger
A slave node of Apollo's Redis Cluster failed and needed to be replaced.

Detection

–

Remediation

–

Action items

–

Lessons

Having several replicas did not protect Apollo, because the client side, not the Redis Cluster server, caused the failure. Grab suggests three ways to avoid it: a separate Redis Cluster client used only for pipelining with RouteByLatency set to true, so pipelined reads go through the master nodes when latency is under 1ms, watching master CPU because they take more load; checking the error of each command in a pipeline instead of failing the whole call, so successful queries are not affected; and changing the Go-Redis library so that the pipeline path reloads the nodes' state when an error happens, as defaultProcessor already does.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.