Incident Library

IL-0292 Google Cloud · 2017-11-06gcpdatabase

App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read

App Engine Memcache needs a globally consistent view of the datacenter that currently serves each application, so that it stays strongly consistent when traffic fails over to another datacenter. The configuration that maps applications to datacenters is stored in a global database. After a configuration update, the database entity that holds this configuration became unavailable for both reads and writes. Memcache treats the configuration as invalid if it cannot be refreshed within 20 seconds, so when clients could not fetch it, Memcache stopped working in every region from 12:33 to 14:23 US/Pacific, 1 hour and 50 minutes.

An attempt at 12:59 to roll back the latest configuration change failed, because the rollback was itself an update to the configuration in the same global database. At 14:21 engineers managed to update the configuration by sending the update request with a long enough deadline, which made all replicas of the database synchronise and let clients read the mapping again. The status page later put full recovery for all projects at 15:30. As a temporary mitigation Google reduced the number of readers of the global configuration, which avoids the contention during writes that had led to the unavailability, and started work to regionalise the configuration to limit the blast radius of similar failures.

Severity
–
Scope
App Engine Memcache unavailable in all regions; all HTTP requests and App Engine API calls failed on Managed VMs; higher Datastore latency and errors for some customers
Users affected
All projects using Memcache
Region
All regions
Time to detect
1m
Time to resolve
1h 50m

Symptoms

From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

While Memcache was down, some customers saw higher Datastore latency and errors. Google attributes these to applications sending surges of Datastore operations while Memcache was unavailable, which can cause contention or hotspotting on particular entities or key ranges. Customers on Managed VMs saw every HTTP request and App Engine API call fail, while App Engine Flexible Environment, the successor to Managed VMs, was not affected.

Engineers first looked for recent changes and at 12:59 tried to revert the latest change to the configuration file, but the rollback needed a write to the same global database, and that write also failed. When Memcache came back at 14:23, a surge of traffic raised the load on Datastore servers; some US applications saw higher get latency until 14:31 and higher put latency until 15:04, and Memcache traffic was ramped up gradually while the error rate fell.

Symptom Service outageDatabase overloadHigh latencyHigh error rate

Root cause

Root cause Configuration change

Component In-memory cacheDatabase
Contributing factor Ineffective rollbackExcessive blast radius
Trigger
A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

Detection

Google received an automated alert at 12:34 US/Pacific, one minute after the start.

Remediation

Action items

Lessons

–

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, In-memory cache, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.