IL-0143 Google Cloud · 2019-11-04kubernetesgcp
From 4 November 2019 some GKE nodes running one of thirteen 1.13 and 1.14 patch versions had more kernel panics than normal after being upgraded. While a fix was prepared, Google advised affected users to go back to an earlier patch release not on the affected list, and users on a Release Channel to contact support to downgrade their nodes.
Fixed versions 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12 became available on 11 November and were rolled out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves. The fix downgraded the Container-Optimized OS image to cos-73-11647-293-0 as a temporary mitigation, and Google expected the next GKE release to bring an upgraded kernel with a fix for the panics. The incident was closed on 13 November at 15:38 US/Pacific.
Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.
The affected GKE versions were 1.13.11-gke.9, 1.14.7-gke.14, 1.13.12-gke.1, 1.14.8-gke.1, 1.13.11-gke.11, 1.13.12-gke.2, 1.14.7-gke.17, 1.14.8-gke.2, 1.13.12-gke.3, 1.14.8-gke.6, 1.13.11-gke.12, 1.13.12-gke.4 and 1.14.8-gke.7. On 7 November the incident was downgraded to an orange Service Disruption because very few projects were actually affected.
Symptom Node kernel panic
Root cause Software defect
–
Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared Node operating system image, Linux kernel
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020same root cause · shared Node operating system image, Linux kernel
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025same root cause · shared Node operating system image
IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Linux kernel
IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNIMicrosoft Azure · 2020same root cause · shared Node operating system image
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020same root cause · shared Linux kernel