Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Stale endpoint routing

오래된 엔드포인트 라우팅 · stale-service-routing

9 cases · 2017 to 2025

Root cause Software defect5 DNS or network fault2 Operator error1 Breaking change in an upgrade1

Component Ingress controller6 Cloud load balancer3 conntrack table3 kube-proxy3

Contributing factor Unreviewed defaults3 Monitoring blind spot2 Unverified assumption1 Automation amplification1

Organisation Cisco ThousandEyes1 Cresta1 Honeycomb1 Lotte E-commerce1 Monzo

Cases

IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeoutCisco ThousandEyes · 2025
Root cause Software defect
IL-0081 API requests timed out during deployment rollouts because the ingress-nginx enable-serial-reloads option held back endpoint updates while a long NGINX reload was runningCresta · 2025
Root cause Software defect
A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced

View on the graph

Kubernetes API server2

Pod termination2

Credential broker for pods1

Horizontal pod autoscaler1

Persistent volumes and stateful sets1

Change with a delayed effect1

Ignored early signal1

Harmful manual action1

No safeguard on destructive change1

Overlapping failures1

Ineffective rollback1

Single point of failure1

Unrepresentative staging1

1

UK Ministry of Justice1

IL-0203

Honeycomb · 2024

Root cause Breaking change in an upgrade

IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022

Root cause DNS or network fault

IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020

Root cause Operator error

IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020

Root cause DNS or network fault

IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them trafficRavelin · 2019

Root cause Software defect

IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its stateNordstrom · 2017

Root cause Software defect

IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outageMonzo · 2017

Root cause Software defect