Cases9 cases

2 to 6 of 9

1. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space

   Adevinta2024

   Symptoms

   One evening the platform team that runs SCHIP, Adevinta's internal runtime platform on Kubernetes, was paged because some DaemonSets were failing. Checking the pod status showed this error from the network plugin: name="aws-cni" failed (add): add cmd: failed to assign an IP address to container

   IP address exhaustionPods Pending

   Trigger

   Not stated for the first page. It came in June, during the migration from SCHIP's kube-aws-based clusters to EKS-based clusters; the write-up mentions spikes of pods, such as a job that goes rogue or failing jobs that keep their IPs, as ways a cluster can run out.

   Root cause

   Capacity exhaustion

   Component

   AWS VPC CNIVPC subnet or pod address rangeKarpenter

   Contributing factor

   Unreviewed defaultsNo capacity planning

   Detection

   A pager alert that some DaemonSets were failing. The on-call engineer checked the pod status and found the aws-cni error about failing to assign an IP address.

   Remediation

   • Rolled out VPC CNI custom networking with a secondary CIDR to the smaller clusters from late August
   • Paused the EKS migration in September while available IPs were very low
   • Rolled out custom networking to the rest of the fleet
   • Added the --reserved-enis parameter to the Karpenter deployment

   Action items

   • Use VPC CNI custom networking with a secondary CIDR across the whole fleet
   • Make Karpenter account for custom networking with the --reserved-enis parameter

   Lessons

   For a platform team, the VPC CNI is not ready to scale in production without time spent on capacity planning, and every option has its pros and cons. For large clusters started today, IPv6 is the cheaper and more future-proof option. Do not assume that what works in small clusters will keep working when they grow.

   View case
2. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours

   CoW Protocol2024

   Symptoms

   At 16:40 UTC on 19 April 2024 CoW Protocol's mainnet production environment stopped producing settlements; the outage officially began then.

   Pods PendingIP address exhaustionPod evictionService outage

   Trigger

   During a cluster capacity expansion, the Infrastructure as Code automation updated the AWS VPC CNI plugin, because the CNI version was not hardcoded in the IaC and the IaC library version had recently changed.

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   Unpinned versionOverlapping failures

   Detection

   The CNI version mismatch was identified as the cause at 17:03 UTC, 23 minutes after settlements stopped.

   Remediation

   • upgraded the Kubernetes cluster
   • set up new node groups with a new version of the networking plugin

   Lessons

   Versions in IaC scripts should be specified explicitly so an automation run cannot update a component such as the CNI plugin unintentionally. Changes in production need change management that is easy to revert.

   View case
3. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
4. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating

   UK Ministry of Justice2023

   Symptoms

   At 08:15 a user posted in the #ask-cloud-platform channel that new deployments were not starting: their pods stayed in ContainerCreating.

   Pods stuck in ContainerCreatingIP address exhaustion

   Root cause

   Configuration change

   Component

   AWS VPC CNINode pool

   Contributing factor

   No configuration tests

   Detection

   A user reported new deployments stuck in ContainerCreating in the #ask-cloud-platform channel at 08:15; the incident was declared at 09:31.

   Remediation

   • Added new nodes and cordoned and drained the six affected nodes
   • Enabled prefix delegation in the VPC CNI on the live cluster after comparing its CNI settings with a 1.23 test cluster

   Action items

   • Add a test or check that confirms IP address allocation is working as expected (#4669)

   Lessons

   The write-up states no lessons beyond its review action, which is to add a check that IP address allocation works.

   View case
5. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case

By root cause4