Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed
    Autumn2026
    Symptoms

    On Sunday March 15, 2026 at about 15:00 GMT, right after collation changes were applied to the customer_entitlements and customer_products tables, database CPU on Autumn's production Postgres spiked and queries began to fail. About 10% of API requests to Autumn failed until 16:15 GMT.

    Database overloadHigh error rateLock contention or deadlock
    Trigger

    A migration that aligned column collations across related tables changed the collation of customer_entitlements and then customer_products on the production database.

    Root cause
    Configuration change
    Component
    Database
    Contributing factor
    Ineffective rollback
    Remediation
    • Shut down all application connections to the database
    • Reverted the collation changes
    • Ran ANALYZE to rebuild the planner statistics and restarted
    Action items
    • Do all future column and schema migrations with the duplicate-dual-write-cutover pattern: add a new column, dual-write, backfill in small batches, create indexes concurrently, then cut over
    • Add a status page
    • Consider making the check and track SDKs fail open by default
    View case
  2. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
    Symptoms

    From 06:54 US/Pacific on 8 January 2025, Google Cloud Pub/Sub customers in ten regions could not publish messages or subscribe to them. Internal telemetry alerted Google engineers at 07:03, nine minutes after the impact started.

    Service outageMonitoring outage
    Trigger

    From 06:54 to 07:30 US/Pacific a bad service configuration change, which over-restricted the permission to access the regional metadata database of Cloud Pub/Sub, was rolled out to multiple regions within a short time.

    Root cause
    Configuration change
    Component
    Message brokerDatabase
    Contributing factor
    Unrepresentative stagingNo staged rolloutIneffective rollback
    Detection

    Internal telemetry alerted Google engineers at 07:03 US/Pacific, nine minutes after the impact started. A second internal telemetry alert at 09:07 showed the ordered subscriptions that could not consume their backlog.

    Remediation
    • Rolled back the configuration change that over-restricted access to the metadata database
    • Identified and repaired every affected ordered subscription so the subscriptions received their backlog
    Action items
    • Enforce parity between pre-production and production environments so configuration changes are caught before production
    • Review change management so configuration changes roll out progressively according to their priority
    • Add monitoring that detects ordering metadata inconsistency
    • Fix the bug in Cloud Pub/Sub ordering metadata management that left ordered messages undelivered
    View case
  3. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017
    Symptoms

    From 12:33 US/Pacific on 6 November 2017, the App Engine Memcache service was unavailable to applications in all regions. Google received an automated alert at 12:34, and the status page said all projects using Memcache were affected.

    Service outageDatabase overloadHigh latencyHigh error rate
    Trigger

    A configuration update, after which the database entity holding the mapping of applications to datacenters could no longer be read or written.

    Root cause
    Configuration change
    Component
    In-memory cacheDatabase
    Contributing factor
    Ineffective rollbackExcessive blast radius
    Detection

    Google received an automated alert at 12:34 US/Pacific, one minute after the start.

    Remediation
    • Sent the configuration update with a long enough deadline, which synchronised the database replicas and let clients read the mapping
    • Ramped Memcache traffic back up gradually while the error rate fell
    Action items
    • Reduce the number of readers of the global configuration to avoid contention during writes
    • Regionalise the configuration to limit the blast radius of similar failures
    View case
Check your cluster for this

These 3 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change3 cases

Component
Database3In-memory cache1Message broker1
Contributing factor
Ineffective rollback3Excessive blast radius1No staged rollout1Unrepresentative staging1
Organisation
Google Cloud2Autumn1
  1. IL-0487 Autumn's Postgres CPU spiked and about 10% of API requests failed after a collation change left the main query unindexed
    Autumn2026
  2. IL-0528 Cloud Pub/Sub could not publish or subscribe in ten regions, and some ordered subscriptions stayed stuck for hours
    Google Cloud2025
  3. IL-0292 App Engine Memcache was down in all regions for 1 hour 50 minutes when its global datacenter mapping could not be read
    Google Cloud2017