IL-0057 Prezi · 2019-10-03 (published)kubernetesawsnetworking
When a pod connects to an external service, meaning one with a public IP, the outgoing packet goes through source network address translation (SNAT) on the worker node, set up by iptables rules. A race condition in the Netfilter module sometimes drops SYN packets while new connection entries are inserted into the conntrack table, which leads to connection timeouts of 1 or 3 seconds. A search for "kubernetes slow outgoing connections" led Prezi to a XING article describing exactly these symptoms.
The fix that article proposes, switching the port allocation to fully random, was not usable on EKS: the amazon-vpc-cni-k8s plugin supports AWS_VPC_K8S_CNI_RANDOMIZESNAT=prng, but the iptables version in the official AMI did not. Prezi instead set AWS_VPC_K8S_CNI_EXTERNALSNAT=true, which turns off SNAT on the node altogether. The write-up calls the EKS documentation for that option misleading and points to an open pull request about it.
After the setting was applied on 8/15, latency returned to the baseline from before the proxy migration. Some improvement from 8/10 came only from running more pods on more nodes, so each pod saw less traffic and hit the race less often.
Prometheus alerts fired on tail latency: from 7/24, the p99 response time of a few services went up after Prezi moved its Nginx-based reverse proxy, smartgateway, from dedicated Elastic Beanstalk machines onto shared Kubernetes worker nodes. The bump affected only the tail, so the earlier monitoring of average response times would most certainly have missed it.
The slow requests were in services that make many sequential service-to-service calls, which at Prezi also pass through the proxy. Stress tests reproduced it. AWS X-Ray traces, added for this investigation, showed a call chain smartgateway → prezipage → smartgateway → presentationservice taking 1.3 seconds, of which a full second was spent between smartgateway and presentationservice, while presentationservice itself answered in 12.8 ms.
The extra delay was always exactly 1 second or, less often, 3 seconds. A custom trace attribute carrying Nginx's $upstream_connect_time confirmed that the time went into opening the outgoing TCP connection.
Prometheus alerts on p99 latency fired after the reverse proxy migration.
–
Alerting on tail latency instead of averages caught a problem that would otherwise have gone unnoticed, and tracing turned a vague slowdown into a measured connect delay. Delays of exactly 1 or 3 seconds on new outgoing connections point to dropped SYN packets, and on EKS the SNAT done by the VPC CNI is where to look.
IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack raceMindTickle · 2019same root cause · shared Linux kernel, conntrack table
IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALBLotte E-commerce · 2022same root cause · shared conntrack table
IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the nodeMindTickle · 2020same root cause · shared AWS VPC CNI
IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entriesPreply · 2020same root cause · shared conntrack table
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.