Incident Library
Applied filters Root causeConfiguration changeComponentIn-memory cacheFactorIneffective rollbackClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

This case involves In-memory cache and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
In-memory cacheDatabase
Contributing factor
Excessive blast radiusIneffective rollback
Organisation
Google Cloud
  1. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017