Incident Library

IL-0246 Adyen · 2026-07-08 (published)kubernetesnetworking

New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"

Adyen runs Cilium across more than 100 Kubernetes clusters. Its big data clusters mix very different pods: Trino pods that open a new connection to one of more than 500 HDFS datanodes for each file, up to about 50,000 connections a minute per pod at peak, Spark batch pods that live from a second to a few hours, and CPU-heavy pods with few connections, on machines with 64 cores and 512GB of RAM or 128 cores and 2TB of RAM. The clusters ran Kubernetes 1.31.7 and Cilium 1.16.5. Months earlier Adyen had set cilium_bpf_map_dynamic_size_ratio to 0.0050 in its Helm chart so that connection tracking capacity would grow with memory, which gave the 2TB nodes a TCP connection tracking table of 16 million entries, twice the default of eight million for that amount of memory.

When an endpoint is created for a new pod, the Cilium agent runs scrubIPsInConntrackTable, which walks the whole connection tracking table to remove entries left for the pod's IP by an earlier pod. The walk takes two BPF syscalls per entry and cannot run in parallel, and on these nodes it reached about 200,000 entries a second, so a table of 7 million entries took 35 seconds and a full table of 16 million would take 80 seconds, against a CNI timeout of 90 seconds. The table held so many entries because the periodic garbage collection of expired entries had almost stopped. Cilium lengthens the interval by 1.5 times whenever a run deletes less than 5% of the entries, up to 12 hours, and on a 16-million-entry table 5% is more than 800,000 entries. A node that first ran light workloads drifted to the 12-hour interval, and when heavy data workloads arrived, expired entries piled up until the next run.

A global mutex on the connection tracking table made concurrent pod starts worse: each pod waited for the scan of the one before it, so the second pod's timer ran out and the rest had no chance. A timed-out create did not stop the agent's work; containerd immediately called DeleteEndpoint(), which also walks the table, and the kubelet's podWorkerLoop retried every 60 to 90 seconds, so create and delete requests queued up faster than they were served. Adyen fixed the interval with conntrackGCInterval: 60s, applied at 9:00 am and rolled out to the DaemonSet by 10:00 am. The table size dropped and stayed stable, endpoint API call durations returned to normal, and the timeout has not appeared since. The clusters serve only analytics and big data, the issue was resolved before any SLO was breached, and real-time payment processing was not affected.

Severity
–
Scope
New pods on affected nodes of Adyen's production big data Kubernetes clusters could not start; no SLO was breached and payment processing was not affected
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On Adyen's production big data Kubernetes clusters, new pods on some nodes could not start because setting up their network through Cilium timed out. Once one pod had timed out, every other pod trying to start on the same node timed out as well.

The pods failed with: Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "0fecf4844d3f8f2df218f09d91f9698bb424e2166551952f46fd7638f4757cf2": plugin type="cilium-cni" failed (add): unable to create endpoint: Cilium API client timeout exceeded. On the affected nodes the duration of the Cilium agent API calls DELETE /v1/endpoint and PUT /v1/endpoint, which add and delete Cilium endpoints, grew linearly over time from 4:20 pm, a sign that endpoint creation and deletion never finished.

The Cilium agent logs showed no related timeouts, even with debug logging turned on. On a broken node `cilium endpoint list` timed out, `cilium endpoint log <ENDPOINT_ID>` showed the endpoint stuck in the regenerating phase, `kubectl get ciliumendpoint` showed it still regenerating while its Kubernetes manifest showed a ready status, and /var/run/cilium/state held endpoint folders with a _next suffix, meaning they had not completed. This placed the problem in the Cilium agent rather than in the kubelet or the CNI plugin.

Execution traces from pprof, taken right after spawning 50 pods on a broken node, showed goroutines spending all their time in BPF syscalls, nextKey() and lookup() in a tight loop at about 426,171 syscalls a second, inside the function scrubIPsInConntrackTable. `cilium status --verbose` showed a TCP connection tracking map of 16777216 entries, and `cilium bpf ct list global` listed about 7 million entries, most of them already expired, some by several hours. Garbage collection metrics showed frequent runs that deleted almost nothing. A goroutine dump taken with gops showed 57 goroutines in createEndpoint() waiting on an RWMutex and 56 in scrubIPsInConntrackTable() waiting for a lock, and the metric cilium_datapath_conntrack_gc_duration_seconds had risen from 1s to 80s.

Symptom Pods stuck in ContainerCreating

Root cause

Root cause Configuration change

Component Cilium
Contributing factor Change with a delayed effectUnreviewed defaultsRetry stormNo alert on limit
Trigger
Heavy data workloads landed on nodes where the interval of Cilium's periodic garbage collection of expired connection tracking entries had grown, during a quiet period, toward its 12-hour maximum.

Detection

–

Remediation

Action items

Lessons

Scaling parameters can interact with other logic months later as traffic grows: the larger table that solved capacity limits on 512GB nodes interacted with the garbage collection auto-scaling on the 2TB nodes. Logs only showed symptoms, so profiling and tracing across the container runtime, the CNI plugin, the Cilium agent and the kernel were needed, and simple arithmetic with measured numbers pointed at the cause early. Auto-scaling algorithms need bounds, and the 12-hour maximum interval was too long for this workload. A timeout protects the caller but does not cancel the work behind it. Adyen recommends that Cilium deployments with dynamic workloads alert on cilium_datapath_conntrack_gc_duration_seconds and cilium_datapath_conntrack_gc_entries and set conntrackGCInterval: 60s.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cilium among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.