Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019
    Symptoms

    Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.

    Node kernel panic
    Trigger

    Upgrading GKE node pools to one of the affected 1.13 or 1.14 versions.

    Root cause
    Software defect
    Component
    Node operating system imageLinux kernel
    Contributing factor
    –
    Remediation
    • advised affected users to downgrade to a patch release not on the affected list, with support helping Release Channel users
    • released GKE 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12, which downgraded Container-Optimized OS to cos-73-11647-293-0
    • rolled the fixed versions out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves
    Action items
    • Ship a GKE release with an upgraded kernel that fixes the panics
    Lessons

    Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.

    View case
Check your cluster for this

These 6 cases involve Node operating system image and Linux kernel. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect6 cases

Component
Node operating system image6Linux kernel3Azure CNI1Container CPU limit1GPU driver1Karpenter1kubelet1Node launch template1Node pool1Terraform1AWS VPC CNI1
Contributing factor
Automation amplification3Unverified assumption2Cache-masked failure1Unpatched known bug1Monitoring blind spot1No health probes1No staged rollout1Overlapping failures1
Organisation
Buffer1DFDS1Google Cloud1Inflab1Microsoft1Microsoft Azure1
  1. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  2. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft2025
  3. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  4. IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI
    Microsoft Azure2020
  5. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit
    Buffer2020
  6. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019