Incident Library
Applied filters Root causeSoftware defectComponentNode operating system imageSymptomNode kernel panicClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019
    Symptoms

    Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.

    Node kernel panic
    Trigger

    Upgrading GKE node pools to one of the affected 1.13 or 1.14 versions.

    Root cause
    Software defect
    Component
    Node operating system imageLinux kernel
    Contributing factor
    –
    Remediation
    • advised affected users to downgrade to a patch release not on the affected list, with support helping Release Channel users
    • released GKE 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12, which downgraded Container-Optimized OS to cos-73-11647-293-0
    • rolled the fixed versions out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves
    Action items
    • Ship a GKE release with an upgraded kernel that fixes the panics
    Lessons

    Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.

    View case
Check your cluster for this

This case involves Linux kernel and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Linux kernelNode operating system image
Contributing factor
–
Organisation
Google Cloud
  1. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image
    Google Cloud2019