Incident Library

IL-0278 Redis · 2016-02-08databasegcp

A weekly update put a Redis Cloud management process into an infinite loop, and most nodes of a GCP cluster failed

Redis Cloud clusters run Redis's cluster tools on bare-metal or virtual servers, each cluster hosting many isolated databases, and are built to survive the failure of a single node. Each week the DevOps team deploys an upgrade of non-critical cluster components, first to one cluster with sanity and stability testing and then one cluster at a time, cloud by cloud; the week's rollout began at 10:30 UTC. At 17:30 UTC it reached the GCP cluster us-central1-1-1.gce.garantiadata.com, where it exposed a serious bug in one of the cluster's management processes. The process locked into a resource-intensive infinite loop, and after several minutes of thrashing each server became unresponsive. One node failed at 17:41 UTC and failed over automatically; four more stopped responding at 17:45 UTC, and with a majority of nodes down the cluster could not recover by itself.

Between 17:45 and 19:30 UTC the team shut down and terminated all nodes, deleted the cluster's endpoints from its DNS zone, built new nodes from scratch without the weekly update and reconnected the original nodes' storage to them. Databases without persistence were bootstrapped and available at once; databases with persistence were bootstrapped and recovered from storage into memory, taking longer the bigger they were. Throttled DNS deletes whose retries failed had to be cleaned up by hand, which lengthened the downtime. By 19:30 UTC over 95% of affected users had service again, and at 20:38 UTC the last and largest database finished recovering from persistence.

Severity
–
Scope
Redis Cloud databases on the GCP cluster us-central1-1-1.gce.garantiadata.com were unavailable; over 95% of affected users were back by 19:30 UTC and all by 20:38 UTC
Users affected
Redis Cloud users with databases endpoints ending in us-central1-1-1.gce.garantiadata.com
Region
us-central1
Time to detect
–
Time to resolve
2h 53m

Symptoms

On 8 February 2016 at 17:45 UTC, four nodes of the Redis Cloud cluster us-central1-1-1.gce.garantiadata.com on Google Cloud Platform stopped responding, four minutes after a first node had failed. The databases whose endpoints end in us-central1-1-1.gce.garantiadata.com went down.

The first node failed at 17:41 UTC and an automated failover handled it successfully. When four more nodes stopped responding at 17:45 UTC, a majority of the cluster had failed and automatic recovery was no longer possible, so the DevOps team started a manual recovery. During that recovery the DNS provider throttled the delete requests generated by Redis's tools, and the retries of the throttled requests were not handled correctly, leaving DNS entries that had to be removed by hand before the cluster could be relaunched.

Symptom Service outageFailover failure

Root cause

Root cause Faulty deployment

Component In-memory cacheDatabase failover managerAuthoritative DNS servers and records
Contributing factor Overlapping failures
Trigger
The weekly upgrade of Redis Cloud's non-critical cluster components reached its first Google Cloud Platform cluster, us-central1-1-1.gce.garantiadata.com, at 17:30 UTC.

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Database failover manager, Authoritative DNS servers and records among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.