Cases11 cases

10 to 11 of 11

1. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets

   loveholidays2020

   Symptoms

   Shortly after a round of changes, loveholidays saw connectivity problems and raised error rates on its aggregation service. The service is a proxy between two other services of more than 100 instances each, handles over 10,000 requests per second continuously and can spike to four times that.

   Connection resetsHigh error rateIntermittent connection failures

   Trigger

   Three changes together: a change in service architecture, halving the RAM of the nodes that run the aggregation service, and sharding a downstream service, which doubled the number of connections to track.

   Root cause

   Capacity exhaustion

   Component

   conntrack tablekube-proxyNode poolHAProxy

   Contributing factor

   Unknown limitNo capacity planningNo alert on limit

   Detection

   Rising error rates and climbing connection reset errors in HAProxy, then node_exporter conntrack metrics compared with their limit.

   Remediation

   • moved the aggregation service to nodes with more RAM
   • limited the number of aggregation service pods per node

   Action items

   • Alert when conntrack table use goes above 75%, with (node_nf_conntrack_entries / on (pod) node_nf_conntrack_entries_limit / on (pod) group_right kube_pod_info) > 0.75

   Lessons

   The conntrack limit is tied to node memory, so resizing a node pool silently resizes the connection table, and on GKE the ratio did not match the commonly cited formula. A high-traffic proxy on small nodes can exhaust it, and the symptoms are connection resets and timeouts, so conntrack use needs its own metric and alert.

   View case
2. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing

   loveholidays2020

   Symptoms

   A colleague wrote in Slack that his application was taking a long time to deploy. A deployment that normally finishes in minutes had been running for about an hour: about half of the new pods were running and serving traffic, and the other half were Pending.

   Pods PendingIP address exhaustionScale-up failure

   Trigger

   The cluster reaching 256 nodes, the most its /16 pod secondary range could hold at 256 addresses per node; every further node creation failed.

   Root cause

   Capacity exhaustion

   Component

   VPC subnet or pod address rangeCluster autoscalerNode pool

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   A colleague reported in Slack that a deployment was slow; the pending pods' events and the cluster autoscaler status ConfigMap showed the exhausted IP space.

   Remediation

   • lowered the maximum pods per node on selected node pools, cutting IP consumption by about 30%
   • moved some node pools to larger instance types

   Action items

   • Lower the maximum pods per node on node pools that do not run many pods
   • Use larger instance types in some node pools

   Lessons

   In a VPC-native GKE cluster the number of nodes, not the number of pods, uses up the pod range: by default each node reserves 256 addresses, so even a /16 is full at 256 nodes. Maximum pods per node and node size are the practical levers; larger nodes pack better but take more pods down when they fail.

   View case

By root cause1