Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Linux kernel

리눅스 커널 · linux-kernel

12 cases · 2019 to 2025

Root cause Software defect5 Capacity exhaustion2 DNS or network fault2 Breaking change in an upgrade2 Configuration change1

Contributing factor Unpatched known bug4 Unreviewed defaults3 Monitoring blind spot3 Unverified assumption2

Symptom Intermittent timeouts5 High latency4 Node NotReady4 Service outage4

Organisation Altinity1 Barush Mendez1 Buffer1 DFDS1 DigitalOcean1

Cases, by severity

IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025
Root cause Breaking change in an upgrade
IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hoursGitHub · 2025
Root cause Breaking change in an upgrade

View on the graph

Automation amplification2

Change with a delayed effect2

No staged rollout2

Single point of failure2

Ignored early signal1

Limited debug access1

No capacity planning1

No health probes1

No resource requests or limits1

Overlapping failures1

High error rate3

Intermittent connection failures3

CPU throttling2

Monitoring outage2

Probe failures2

Control plane unresponsive1

Container crash loop1

DNS resolution failures1

Sustained high node CPU1

Node kernel panic1

IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days

Barush Mendez · 2025

Root cause Software defect

IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networkingDigitalOcean · 2023

Root cause Configuration change

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020

Root cause Software defect

IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999LINE · 2020

Root cause Capacity exhaustion

IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020

Root cause Software defect

IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019

Root cause Software defect

IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019

Root cause DNS or network fault

IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack raceMindTickle · 2019

Root cause DNS or network fault

IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was downMoonlight · 2019

Root cause Capacity exhaustion