Cases9 cases

9 to 9 of 9

1. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race

   Prezi2019

   Symptoms

   Prometheus alerts fired on tail latency: from 7/24, the p99 response time of a few services went up after Prezi moved its Nginx-based reverse proxy, smartgateway, from dedicated Elastic Beanstalk machines onto shared Kubernetes worker nodes. The bump affected only the tail, so the earlier monitoring of average response times would most certainly have missed it.

   High latencyIntermittent timeouts

   Trigger

   Moving the reverse proxy onto shared EKS worker nodes, where its outgoing connections went through source NAT on the node and some SYN packets were dropped by a race in Netfilter's conntrack.

   Root cause

   DNS or network fault

   Component

   conntrack tableAWS VPC CNILinux kernelNode operating system image

   Contributing factor

   Unreviewed defaultsUnpatched known bug

   Detection

   Prometheus alerts on p99 latency fired after the reverse proxy migration.

   Remediation

   • added AWS X-Ray tracing and an $upstream_connect_time trace attribute to locate the delay
   • ran more pods of the service on more nodes, which reduced but did not end the delays
   • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true on the AWS VPC CNI to turn off SNAT on the nodes

   Lessons

   Alerting on tail latency instead of averages caught a problem that would otherwise have gone unnoticed, and tracing turned a vague slowdown into a measured connect delay. Delays of exactly 1 or 3 seconds on new outgoing connections point to dropped SYN packets, and on EKS the SNAT done by the VPC CNI is where to look.

   View case

By root cause4