Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025sev1
    Symptoms

    Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

    Scale-up failureContainer crash loopService outageDNS resolution failures
    Trigger

    Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

    Root cause
    Breaking change in an upgrade
    Component
    GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets
    Contributing factor
    Deferred upgradesSingle point of failureIneffective rollbackNo headroom
    Remediation
    • Created a new GKE cluster at v1.33, the version of the forced upgrade
    • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
    • Updated the cert-manager configuration for Kubernetes 1.33
    • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
    • Decommissioned the broken cluster
    Action items
    • Configure all clusters to use the GKE stable release channel
    • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
    • Review spot instance usage for critical workloads and add on-demand fallback pools
    • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
    • Document a Kubernetes, Istio and cert-manager compatibility matrix
    • Create an automated upgrade testing pipeline
    • Reserve static IPs for quick migrations
    • Pin versions of Istio, cert-manager, CSI and DNS addons
    • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines
    Lessons

    GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

    View case
  2. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025sev1
    Symptoms

    The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

    Service outageContainer crash loopPersistent volume full
    Trigger

    A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

    Root cause
    Capacity exhaustion
    Component
    Persistent volumes and stateful setsIn-memory cache
    Contributing factor
    No capacity planning
    Remediation
    • backed off the Redis StatefulSet and the deployment
    • deleted the corrupted PVC and rebuilt it
    • restarted the StatefulSet and the deployment
    Action items
    • Increase the Klamm Redis cache volume to 100Mi
    • Integrate Sysdig monitoring on the namespace
    • Enable replica pods on the Redis cluster
    View case
Check your cluster for this

These 2 cases involve Persistent volumes and stateful sets and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Breaking change in an upgrade1 case

Component
ConfigMaps and SecretsClient-side DNS resolver and cacheGKE control planeNode poolPersistent volumes and stateful sets
Contributing factor
No headroomIneffective rollbackSingle point of failureDeferred upgrades
Organisation
Wealthy
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025

Capacity exhaustion1 case

Component
In-memory cachePersistent volumes and stateful sets
Contributing factor
No capacity planning
Organisation
BC Government
  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025