Cases3 cases

1 to 3 of 3

1. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"

   Adyen2026

   Symptoms

   On Adyen's production big data Kubernetes clusters, new pods on some nodes could not start because setting up their network through Cilium timed out. Once one pod had timed out, every other pod trying to start on the same node timed out as well.

   Pods stuck in ContainerCreating

   Trigger

   Heavy data workloads landed on nodes where the interval of Cilium's periodic garbage collection of expired connection tracking entries had grown, during a quiet period, toward its 12-hour maximum.

   Root cause

   Configuration change

   Component

   Cilium

   Contributing factor

   Change with a delayed effectUnreviewed defaultsRetry stormNo alert on limit

   Remediation

   • Set a fixed connection tracking garbage collection interval, conntrackGCInterval: 60s, and rolled it out to the Cilium DaemonSet

   Action items

   • Alert on the connection tracking garbage collection duration and table size metrics

   Lessons

   Scaling parameters can interact with other logic months later as traffic grows: the larger table that solved capacity limits on 512GB nodes interacted with the garbage collection auto-scaling on the 2TB nodes. Logs only showed symptoms, so profiling and tracing across the container runtime, the CNI plugin, the Cilium agent and the kernel were needed, and simple arithmetic with measured numbers pointed at the cause early. Auto-scaling algorithms need bounds, and the 12-hour maximum interval was too long for this workload. A timeout protects the caller but does not cancel the work behind it. Adyen recommends that Cilium deployments with dynamic workloads alert on cilium_datapath_conntrack_gc_duration_seconds and cilium_datapath_conntrack_gc_entries and set conntrackGCInterval: 60s.

   View case
2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
3. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating

   UK Ministry of Justice2023

   Symptoms

   At 08:15 a user posted in the #ask-cloud-platform channel that new deployments were not starting: their pods stayed in ContainerCreating.

   Pods stuck in ContainerCreatingIP address exhaustion

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   No configuration tests

   Detection

   A user reported new deployments stuck in ContainerCreating in the #ask-cloud-platform channel at 08:15; the incident was declared at 09:31.

   Remediation

   • Added new nodes and cordoned and drained the six affected nodes
   • Enabled prefix delegation in the VPC CNI on the live cluster after comparing its CNI settings with a 1.23 test cluster

   Action items

   • Add a test or check that confirms IP address allocation is working as expected (#4669)

   Lessons

   The write-up states no lessons beyond its review action, which is to add a check that IP address allocation works.

   View case

By root cause1