Incident Library
Applied filters Root causeBreaking change in an upgradeSymptomContainer crash loopDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025sev1
    Symptoms

    Nodes that already existed kept running, but every new node, spot or standard, on either v1.32 or v1.33, failed to initialise and could not join the cluster. Spot nodes that were preempted and recreated never came back, so services degraded progressively: background workers on spot instances, reporting services and order allocation stopped completely, and the APIs on existing nodes kept answering but could not scale for load spikes.

    Scale-up failureContainer crash loopService outageDNS resolution failures
    Trigger

    Google upgraded the GKE control plane from v1.32.x to v1.33.2 at the version deprecation deadline, although auto-upgrade was disabled on the node pools.

    Root cause
    Breaking change in an upgrade
    Component
    GKE control planeNode poolClient-side DNS resolver and cachePersistent volumes and stateful setsConfigMaps and Secrets
    Contributing factor
    Deferred upgradesSingle point of failureIneffective rollbackNo headroom
    Remediation
    • Created a new GKE cluster at v1.33, the version of the forced upgrade
    • Upgraded Istio to 1.27.1 for Kubernetes 1.33 compatibility
    • Updated the cert-manager configuration for Kubernetes 1.33
    • Migrated all production workloads to the new cluster and verified pod scheduling, DNS resolution, PVC mounting and external integrations
    • Decommissioned the broken cluster
    Action items
    • Configure all clusters to use the GKE stable release channel
    • Implement a separate management cluster for Argo CD, monitoring and other control components (Q4 2025)
    • Review spot instance usage for critical workloads and add on-demand fallback pools
    • Document the SRE upgrade procedure: management cluster first, production after a monitoring period of several days
    • Document a Kubernetes, Istio and cert-manager compatibility matrix
    • Create an automated upgrade testing pipeline
    • Reserve static IPs for quick migrations
    • Pin versions of Istio, cert-manager, CSI and DNS addons
    • Stay current with stable releases and regularly check GKE alerts and announcements to avoid deprecation deadlines
    Lessons

    GKE upgrades the control plane at a version's deprecation deadline even when auto-upgrade is disabled, so staying inside the supported version window is how to avoid a forced upgrade. A control plane upgrade can leave existing nodes healthy while every new node fails, and spot nodes that cannot rejoin after preemption turn this into cascading failures. Relying heavily on spot instances for critical workers turned a node join failure into a progressive outage. Kubernetes upgrades have to be planned together with dependent components such as Istio and cert-manager, and without a standby cluster on the same provider there was no immediate failover; the only recovery left was building a new cluster.

    View case
  2. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024
    Symptoms

    On 24 October 2024 every BSEE SaaS customer instance showed the Environment Unavailable message, and the SaaS Uptime Over Time metric dropped sharply at 04:45. The Technology Team learned of the outage when the Growth team posted about it at 08:07 and a member of the Product team told the Technology Team verbally at 08:15.

    Container crash loopService outage
    Trigger

    AWS made Bottlerocket v1.26.0 the latest Bottlerocket AMI in eu-west-1 between 02:00 and 04:30, and at 04:00 Karpenter saw the new version and started replacing the EC2 nodes of the production EKS cluster with it.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imageKarpenter
    Contributing factor
    Unpinned version
    Detection

    The Growth team reported the outage in the BSEE cloud onboarding issues channel at 08:07, and a member of the Product team told the Technology Team verbally at 08:15. The SaaS Uptime Over Time metric had already dropped at 04:45.

    Remediation
    • Reverted all EC2 instances in the SaaS production account to Bottlerocket v1.25.0
    • Deployed Bottlerocket 1.25.0 across all EKS clusters and rolled out new EKS nodes
    Action items
    • Change the Karpenter resources to deploy a specific versioned Bottlerocket AMI instead of the latest released AMI
    • Add the Bottlerocket release ATOM feed to an internal tooling release dashboard
    • Set up a public status page to communicate with customers
    View case
Check your cluster for this

These 2 cases involve ConfigMaps and Secrets and Client-side DNS resolver and cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade2 cases

Component
ConfigMaps and Secrets1Client-side DNS resolver and cache1GKE control plane1Karpenter1Node operating system image1Node pool1Persistent volumes and stateful sets1
Contributing factor
No headroom1Ineffective rollback1Single point of failure1Deferred upgrades1Unpinned version1
Organisation
PortSwigger1Wealthy1
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  2. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024