IL-0017 UK Ministry of Justice · 2023-07-21kubernetesaws
On 21 July 2023 new pods on the Ministry of Justice Cloud Platform live cluster stayed in ContainerCreating because the AWS VPC CNI was not allocating IP addresses on six nodes. The team added new nodes and began to cordon and drain the affected ones.
The underlying cause was a setting missing on the live cluster. Comparing the CNI configuration of a 1.23 test cluster with live showed the difference, and at 12:42 the team ran the command that enables prefix delegation on the live cluster, which ended the incident. Time to repair and time to resolve were both 4h 27m. The single review action is a test or check that confirms IP address allocation works as expected.
At 08:15 a user posted in the #ask-cloud-platform channel that new deployments were not starting: their pods stayed in ContainerCreating.
From 09:00 the Cloud Platform team put together a list of every affected namespace, and it declared an incident at 09:31. By 09:45 the team had traced the stuck pods to six nodes on which the VPC CNI was not allocating IP addresses to new pods.
Adding fresh nodes and cordoning and draining the six affected nodes moved workloads off them, but did not explain the fault. The cause was found only at 12:35, about three hours later, when the team compared the CNI settings of a 1.23 test cluster with those of the live cluster and saw that one setting differed.
Symptom Pods stuck in ContainerCreatingIP address exhaustion
Root cause Configuration change
A user reported new deployments stuck in ContainerCreating in the #ask-cloud-platform channel at 08:15; the incident was declared at 09:31.
The write-up states no lessons beyond its review action, which is to add a check that IP address allocation works.
IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024same root cause · shared Node pool, AWS VPC CNI
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023same root cause · shared Node pool, AWS VPC CNI
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodesDFDS · 2023 · sev1same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared AWS VPC CNI
IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a clusterTarget · 2018same root cause · shared Node pool