Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Component

conntrack table

conntrack 테이블 · conntrack

8 cases · 2019 to 2025

Root cause DNS or network fault4 Capacity exhaustion2 Configuration change1 Software defect1

Contributing factor Unreviewed defaults5 Unverified assumption1 Automation amplification1 Excessive blast radius1

Symptom DNS resolution failures3 High error rate3 High latency3 Intermittent timeouts3

Organisation Backpackr1 Cisco ThousandEyes1 LINE+1 Lotte E-commerce1 loveholidays

Cases, by severity

IL-0104 After a CoreDNS pod was OOMKilled, the talk names three causes: Istio's aggressive DNS cache refresh on one reused client port, kube-proxy not cleaning up conntrack entries of terminated pods, and the 2-minute UDP conntrack timeoutCisco ThousandEyes · 2025
Root cause Software defect
IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025
Root cause Capacity exhaustion

View on the graph

Change with a delayed effect1

Ignored early signal1

Unpatched known bug1

Monitoring blind spot1

No alert on limit1

No capacity planning1

Single point of failure1

Stale endpoint routing3

Connection resets2

Intermittent connection failures2

Database overload1

Service outage1

1

IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

Lotte E-commerce · 2022

Root cause DNS or network fault

IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on NginxLINE+ · 2022

Root cause Configuration change

IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020

Root cause DNS or network fault

IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020

Root cause Capacity exhaustion

IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack racePrezi · 2019

Root cause DNS or network fault

IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack raceMindTickle · 2019

Root cause DNS or network fault