IL-0019 UK Ministry of Justice · 2023-09-18kubernetesaws
On 18 September 2023 the Ministry of Justice Cloud Platform team upgraded its EKS Terraform module to version 18. The new module version changed the node launch template so that new nodes got a 20 GB root volume instead of 100 GB. As workloads were drained onto the new nodes, several nodes ran short of disk space, image pulls failed with "no space left on device", and numerous pods were in the Evicted and ContainerStateUnKnown states.
At 15:34 the team uncordoned the old default node group and at 15:35 started draining the new nodes to move workloads back. The incident was repaired at 17:54, 4h 12m after the first alert. On 19 September the team validated the fix and worked through the launch template changes, and on 20 September applied the fix to the manager cluster and then the live cluster and started draining the old node group again.
That second drain ran into pod IP allocation: more pods sat in ContainerCreating, with "failed to assign an IP address to container" ENI errors, which the team thought might be IP prefix starvation that eases as draining the old nodes frees prefixes. At 19:18 on 20 September all nodes were drained, no pods were in an error state and the disk space issue was resolved; the write-up gives the time to resolve as 35h 36m. The review actions cover comparing launch template changes during module upgrades, drift tests on the launch template, image pull tests at realistic sizes, a disk space configuration check in the RootVolUtilisation alert runbook, scaling CoreDNS with the number of nodes, IPv6 as a way out of prefix starvation, and logging and alerts for the CNI and ipamd.
At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".
The team was in the middle of upgrading the EKS Terraform module to version 18 and draining nodes. At 14:27 numerous pods were in the Evicted and ContainerStateUnKnown states, and the write-up notes that deployments might not be scheduled consistently and might fail. At 15:26 the team compared the disk allocated to an old node with that of a new node and found the new node had only 20 GB.
The second attempt two days later showed a different symptom. While the old node group was being drained on 20 September, more pods than usual sat in ContainerCreating at 15:04, and at 15:25 there were many "failed to assign an IP address to container" ENI errors. The CNI logs said "Unable to get IP address from CIDR: no free IP available in the prefix". The team read this as IP prefix starvation that eased as draining the old nodes freed prefixes.
Root cause Configuration change
The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.
The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.
IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024same root cause · shared AWS VPC CNI, Node pool
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023same root cause · shared AWS VPC CNI, Node pool
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1same root cause · shared Node pool, Node launch template
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared AWS VPC CNI
IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018same root cause · shared Node pool
IL-0017 linked by the editor