Cases12 cases

10 to 12 of 12

1. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race

   Prezi2019

   Symptoms

   Prometheus alerts fired on tail latency: from 7/24, the p99 response time of a few services went up after Prezi moved its Nginx-based reverse proxy, smartgateway, from dedicated Elastic Beanstalk machines onto shared Kubernetes worker nodes. The bump affected only the tail, so the earlier monitoring of average response times would most certainly have missed it.

   High latencyIntermittent timeouts

   Trigger

   Moving the reverse proxy onto shared EKS worker nodes, where its outgoing connections went through source NAT on the node and some SYN packets were dropped by a race in Netfilter's conntrack.

   Root cause

   DNS or network fault

   Component

   conntrack tableAWS VPC CNILinux kernelNode operating system image

   Contributing factor

   Unreviewed defaultsUnpatched known bug

   Detection

   Prometheus alerts on p99 latency fired after the reverse proxy migration.

   Remediation

   • added AWS X-Ray tracing and an $upstream_connect_time trace attribute to locate the delay
   • ran more pods of the service on more nodes, which reduced but did not end the delays
   • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true on the AWS VPC CNI to turn off SNAT on the nodes

   Lessons

   Alerting on tail latency instead of averages caught a problem that would otherwise have gone unnoticed, and tracing turned a vague slowdown into a measured connect delay. Delays of exactly 1 or 3 seconds on new outgoing connections point to dropped SYN packets, and on EKS the SNAT done by the VPC CNI is where to look.

   View case
2. IL-0047 Pod-to-pod requests delayed by 5, 10 or 15 seconds because of DNS lookups lost to a conntrack race

   MindTickle2019

   Symptoms

   Pod-to-pod calls were intermittently slow and the web application felt slow overall. A request that pod A sent at time x reached pod B at x + 5s, x + 10s or x + 15s, always a multiple of five seconds.

   Intermittent timeoutsHigh latencyDNS resolution failures

   Trigger

   The C library resolver sent the A and AAAA queries for each lookup on the same port; where DNAT or SNAT translation takes place, a race condition in conntrack sometimes left one of the queries unanswered, and the resolver waited 5 seconds before retrying.

   Root cause

   DNS or network fault

   Component

   conntrack tableClient-side DNS resolver and cacheLinux kernel

   Contributing factor

   Unreviewed defaults

   Detection

   The team noticed intermittent delays in pod-to-pod calls and slowness in the web application, and that the delays were always multiples of five seconds.

   Remediation

   • added the single-request-reopen option to dnsConfig in the deployment specs
   • set GRPC_DNS_RESOLVER=native in the Dockerfiles of gRPC applications so they used the libc resolver
   • checked with tcpdump port 53 that A and AAAA queries left from different ports

   Lessons

   Delays that are exact multiples of five seconds point to DNS lookups being retried after the resolver's 5-second timeout. A resolver option set in the pod spec only helps processes that use the libc resolver; runtimes with their own resolver, such as gRPC with c-ares, ignore it.

   View case
3. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelApplication database

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case

By root cause5