IL-0112 DFDS · 2022-10-31kubernetesaws
DFDS's Terraform infrastructure-modules were hard-coded to use the latest AWS EKS node AMI as soon as it was available, with no way to pin a specific one, and a daily rollover replaced the cluster's nodes. On 2022-10-31 the rollover picked up AMI v20221027, which was later recalled because of a kernel issue. Nodes on that image had sporadic communication problems: they briefly reported "PLEG is not healthy", sometimes went unhealthy, the number of sockets per node kept growing, and one team saw network timeouts from its pods to other services.
The same day the team upgraded the vpc-cni add-on, the upgrade stalled, and the node flapping was first blamed on it. The team completed the upgrade by cordoning, draining and deleting the stuck nodes one at a time, 15 in total, by 17:35. At 17:21 a GitHub issue about PLEG errors with this AMI led the team to learn that the AMI had been recalled. With no service loss visible, the team decided to monitor and wait for a new AMI release. As socket use kept rising and Kubernetes events reported communication problems, the team decided to roll back on 2022-11-02 at 11:56. A Terraform change that allowed setting a specific AMI was merged at 13:14, the pipeline rolled the nodes back to the previous AMI at 13:46, and at 14:40 the team judged the problem resolved. Time to detection was 7 hours and time to recovery 54 hours 10 minutes.
The write-up lists more factors that made the incident worse or harder to handle: some team applications had no readiness or liveness probes; it happened during an IT conference that most development teams and members of the reporting team itself attended; there were no on-call escalation paths with individual teams; the team that reported problems was hard to get specifics from; upgrading an EKS component on the same day as the new AMI confused the diagnosis; alerting did not catch the NodeNotReady events or the socket growth; and the team lacked deeper knowledge of Kubernetes components.
On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.
From 14:30 some nodes reported NodeNotReady for about 30 seconds at a time and then NodeReady again, with the message "KubeletNotReady PLEG is not healthy". The team assumed this came from the vpc-cni upgrade in progress. Some aws-node pods made progress and then got stuck again, and a terminating pod that was force-deleted was replaced by a new one that never became Ready. Logs for some containers could not be read through the Kubernetes API, although the aws-node containers were running and logging on the nodes themselves.
After the upgrade was finished by hand the NodeNotReady events continued, but no loss of service was seen or reported. On 2022-11-01 at 11:12 the team saw the number of sockets used per node rising continuously. On 2022-11-02 at 10:52 one team reported connectivity problems from some of its pods, network timeouts when talking to other services, but gave no details to investigate.
Root cause Software defect
The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.
When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters.
Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020same root cause · shared Node operating system image, Linux kernel
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020same root cause · shared kubelet, Linux kernel
IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019same root cause · shared Node operating system image, Linux kernel
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node operating system image
IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspacesZenML · 2025same root cause · shared Terraform
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025same root cause · shared Linux kernel
IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico componentGoogle Cloud · 2022same root cause · shared kubelet
IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNIMicrosoft Azure · 2020same root cause · shared Node operating system image
IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018same root cause · shared kubelet
IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018same root cause · shared kubelet