Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

kube-proxy

kube-proxy · kube-proxy

7 cases · 2019 to 2025

Root cause Capacity exhaustion2 DNS or network fault2 Operator error1 Software defect1 Breaking change in an upgrade1

Contributing factor Unreviewed defaults4 Unverified assumption1 Automation amplification1 Ignored early signal1

Symptom DNS resolution failures3 High error rate3 Stale endpoint routing3 Intermittent timeouts2

Organisation Airbnb1 Altinity1 Cisco ThousandEyes1 DevOps Hof1 Lotte E-commerce

Cases, by severity

IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeoutCisco ThousandEyes · 2025
Root cause Software defect
IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025
Root cause Breaking change in an upgrade

View on the graph

Unpatched known bug1

No alert on limit1

No capacity planning1

No staged rollout1

Service outage2

Connection resets1

Intermittent connection failures1

Probe failures1

Load imbalance1

1

IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022

Root cause DNS or network fault

IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020

Root cause DNS or network fault

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019

Root cause Capacity exhaustion

IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx podsDevOps Hof · 2019

Root cause Operator error