IL-0013 loveholidays · 2020-02-15 (published)kubernetesgcpnetworking
Connection tracking (conntrack) is the Linux netfilter feature that records every network connection so that NAT and stateful iptables rules can treat all packets of a connection the same way, and Kubernetes nodes depend on it heavily to route traffic between pods and services. The table has a maximum size, visible in the node_nf_conntrack_entries_limit metric or through sysctl net.ipv4.netfilter.ip_conntrack_max. The commonly cited default is CONNTRACK_MAX = RAMSIZE (in bytes) / 16384 / (x / 32), where x is the pointer size in bits, which makes the limit proportional to node memory. On GKE the team observed something different: roughly 5 conntrack entries per MB of memory.
Either way the limit follows node memory, so halving the RAM of the aggregation nodes shrank the table at the same time as sharding doubled the connections the proxy had to track, and the table filled. The immediate fix was to schedule the aggregation service on nodes with more RAM and to limit how many of its pods run on one node. The team then added an alert on conntrack use above 75%. The write-up also points to a Kubernetes fix expected in version 1.15 or newer (pull request 74840), and to two ways of raising nf_conntrack_max with a privileged sysctl -w net.netfilter.nf_conntrack_max=262144, from an init container or from a DaemonSet on dedicated nodes. It warns that this lets an application manage infrastructure without knowing the node's default, that several applications can keep overwriting each other's value as pods come and go, and that a larger table can use more RAM and add latency.
Shortly after a round of changes, loveholidays saw connectivity problems and raised error rates on its aggregation service. The service is a proxy between two other services of more than 100 instances each, handles over 10,000 requests per second continuously and can spike to four times that.
The application logged errors such as "Caused by: org.apache.http.NoHttpResponseException: xxxxx failed to respond", and HAProxy reported connection reset errors whose count kept climbing.
The team worked through the available networking metrics. Overlaying the node_exporter metric node_nf_conntrack_entries on node_nf_conntrack_entries_limit made it clear that some nodes were running out of conntrack table space. The query used was (node_nf_conntrack_entries / on (pod) node_nf_conntrack_entries_limit / on (pod) group_right kube_pod_info) * 100, which gives the percentage of the conntrack table in use on each pod's node.
Symptom Connection resetsHigh error rateIntermittent connection failures
Root cause Capacity exhaustion
Rising error rates and climbing connection reset errors in HAProxy, then node_exporter conntrack metrics compared with their limit.
The conntrack limit is tied to node memory, so resizing a node pool silently resizes the connection table, and on GKE the ratio did not match the commonly cited formula. A high-traffic proxy on small nodes can exhaust it, and the symptoms are connection resets and timeouts, so conntrack use needs its own metric and alert.
IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool
IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool
IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool
IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025same root cause · shared conntrack table
IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool
IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool
IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 secondsAirbnb · 2019same root cause · shared kube-proxy