Cases17 cases

10 to 14 of 17

1. IL-0445 Chatwork's service on EKS became unstable as conntrack tables on some nodes overflowed and connections were dropped

   Chatwork2021

   Symptoms

   In 2020 Chatwork moved its legacy system from EC2 to EKS, running both sides in parallel with Route53 weighted records. Once more than a certain share of requests was sent to the EKS side, the service suddenly became unstable. On some nodes the conntrack table had gone past its limit and connections to the application were being dropped.

   Intermittent connection failuresDNS resolution failuresIntermittent timeoutsLoad imbalance

   Trigger

   During the migration from EC2 to EKS, Chatwork sent more than a certain share of requests to the EKS side. Later, applying node-local-dns with force_tcp in its configuration made DNS unstable again.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableCoreDNSkube-proxyClient-side DNS resolver and cacheVPC DNS resolver

   Contributing factor

   Unreviewed defaultsUnrepresentative stagingUnknown limit

   Remediation

   • Raised the conntrack maxPerCore value in the kube-proxy configuration
   • Installed dns-autoscaler so CoreDNS ran with more pods
   • Installed node-local-dns as a DaemonSet on every node
   • Replaced force_tcp with prefer_udp in the node-local-dns configuration

   Lessons

   On EKS, dns-autoscaler and node-local-dns made DNS stable, but node-local-dns must not use force_tcp there. Stable DNS also matters for sizing the number of pods correctly, since blocked DNS queries had made the pod count grow.

   View case
2. IL-0056 Calls from EKS pods to services behind an NLB in another VPC failed intermittently because the AWS VPC CNI did source NAT on the node

   MindTickle2020

   Symptoms

   On the Saturday the production cluster was moved from kops to EKS, the platform became very slow and too unstable to keep running, and the team reverted.

   Intermittent connection failuresHigh latency

   Trigger

   Production was moved from a kops cluster using Weave networking to a new EKS cluster using the AWS VPC CNI, without external SNAT enabled.

   Root cause

   DNS or network fault

   Component

   AWS VPC CNI

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team lead saw the platform behaving very slowly after the switch to EKS and phoned the author.

   Remediation

   • reverted production to the kops cluster
   • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true in the AWS VPC CNI

   Action items

   • Monitor and alert on the AWS VPC CNI

   Lessons

   In a networking problem every component in the stack is a suspect; ruling out EKS early cost a lot of time. A DaemonSet that pinged every component from every other one exposed the anomaly that led to the fix. The team had only skimmed the AWS CNI documentation before the migration; knowing how the plugin works internally would have led to better judgements and hypotheses while debugging.

   View case
3. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts

   Omio2020

   Symptoms

   For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

   CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures

   Trigger

   Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelkubelet

   Contributing factor

   Unknown limitMonitoring blind spotUnpatched known bug

   Detection

   Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

   Remediation

   • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters

   Action items

   • Deploy the kernel with the CFS quota bug fix across all clusters
   • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected

   Lessons

   A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

   View case
4. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets

   loveholidays2020

   Symptoms

   Shortly after a round of changes, loveholidays saw connectivity problems and raised error rates on its aggregation service. The service is a proxy between two other services of more than 100 instances each, handles over 10,000 requests per second continuously and can spike to four times that.

   Connection resetsHigh error rateIntermittent connection failures

   Trigger

   Three changes together: a change in service architecture, halving the RAM of the nodes that run the aggregation service, and sharding a downstream service, which doubled the number of connections to track.

   Root cause

   Capacity exhaustion

   Component

   conntrack tablekube-proxyNode poolHAProxy

   Contributing factor

   Unknown limitNo capacity planningNo alert on limit

   Detection

   Rising error rates and climbing connection reset errors in HAProxy, then node_exporter conntrack metrics compared with their limit.

   Remediation

   • moved the aggregation service to nodes with more RAM
   • limited the number of aggregation service pods per node

   Action items

   • Alert when conntrack table use goes above 75%, with (node_nf_conntrack_entries / on (pod) node_nf_conntrack_entries_limit / on (pod) group_right kube_pod_info) > 0.75

   Lessons

   The conntrack limit is tied to node memory, so resizing a node pool silently resizes the connection table, and on GKE the ratio did not match the commonly cited formula. A high-traffic proxy on small nodes can exhaust it, and the symptoms are connection resets and timeouts, so conntrack use needs its own metric and alert.

   View case
5. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic

   Ravelin2019

   Symptoms

   After Ravelin moved its API layer from VMs into its GKE cluster behind a Kubernetes Ingress, the integration tests occasionally received 502 errors.

   Intermittent connection failuresStale endpoint routing

   Trigger

   Pods behind the Ingress being removed by their controller, as happens during rolling deploys and scaling.

   Root cause

   Software defect

   Component

   Ingress controllerPod termination

   Contributing factor

   Unverified assumption

   Detection

   Integration tests occasionally received 502 errors.

   Lessons

   Removing a pod's endpoint and signalling the pod happen at the same time, and an ingress can take a long time to catch up. A pod behind an ingress should keep serving after SIGTERM until its grace period ends, and the grace period should be longer than the load balancer's reprogramming time.

   View case

By root cause6