Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

Node operating system image

노드 운영체제 이미지 · node-image

16 cases · 2019 to 2025

Root cause Software defect6 Breaking change in an upgrade4 Dependency failure3 Configuration change1 Faulty deployment1 DNS or network fault1

Contributing factor Automation amplification6 No staged rollout5 Unreviewed defaults4 Unpatched known bug3

Symptom Node NotReady5 GPU unavailable3 Intermittent timeouts3 Service outage3

Organisation Microsoft Azure3 Google Cloud2 Altinity1 Buffer1 Datadog

Cases, by severity

IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiodTetrate (unnamed U.S. federal customer) · 2025 · sev1
Root cause Dependency failure
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025
Root cause Software defect

View on the graph

Unverified assumption2

Change with a delayed effect2

Limited debug access2

Monitoring blind spot2

Excessive blast radius1

Cache-masked failure1

No health probes1

Overlapping failures1

Scale-up failure2

CPU throttling1

Container crash loop1

DNS resolution failures1

High error rate1

Image pull failures1

Intermittent connection failures1

Monitoring outage1

Node kernel panic1

Node recreation loop1

Probe failures1

Throughput degradation1

1

Tetrate (unnamed U.S. federal customer)1

IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries

Microsoft · 2025

Root cause Software defect

IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025

Root cause Breaking change in an upgrade

IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025

Root cause Breaking change in an upgrade

IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUsGoogle Cloud · 2024

Root cause Dependency failure

IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networkingDigitalOcean · 2023

Root cause Configuration change

IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023

Root cause Breaking change in an upgrade

IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022

Root cause Software defect

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNIMicrosoft Azure · 2020

Root cause Software defect

IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020

Root cause Software defect

IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019

Root cause Software defect

IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019

Root cause DNS or network fault

IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthyMicrosoft Azure · 2019

Root cause Faulty deployment

IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup brokeFREE NOW · 2019

Root cause Dependency failure