Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed
    Autumn2026
    Symptoms

    On Sunday March 15, 2026 at about 15:00 GMT, right after collation changes were applied to the customer_entitlements and customer_products tables, database CPU on Autumn's production Postgres spiked and queries began to fail. About 10% of API requests to Autumn failed until 16:15 GMT.

    Database overloadHigh error rateLock contention or deadlock
    Trigger

    A migration that aligned column collations across related tables changed the collation of customer_entitlements and then customer_products on the production database.

    Root cause
    Configuration change
    Component
    Database
    Contributing factor
    Ineffective rollback
    Remediation
    • Shut down all application connections to the database
    • Reverted the collation changes
    • Ran ANALYZE to rebuild the planner statistics and restarted
    Action items
    • Do all future column and schema migrations with the duplicate-dual-write-cutover pattern: add a new column, dual-write, backfill in small batches, create indexes concurrently, then cut over
    • Add a status page
    • Consider making the check and track SDKs fail open by default
    View case
  2. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 2 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change2 cases

Component
Database2In-memory cache1
Contributing factor
Ineffective rollback2Excessive blast radius1
Organisation
Autumn1Google Cloud1
  1. IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed
    Autumn2026
  2. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017