Incident Library

IL-0449 Enigmo · 2020-12-13 (published)kubernetesaws

Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable

Enigmo upgraded its EKS clusters from 1.13 to 1.15 in spring 2020 and soon began preparing for 1.16. Its node groups had been built from an EC2 LaunchTemplate and AutoScalingGroup following the AWS documentation of the EKS 1.13 era, so each upgrade meant creating new node groups with the new AMI, tainting and draining the old ones and deleting them, three CloudFormation change sets in all. Managed node groups first could not be given security groups, which Pods need to reach RDS, ElastiCache and ALB, but that summer they gained LaunchTemplate support, so Enigmo moved to managed node groups before the 1.16 upgrade. The move itself, creating managed node groups, evicting pods from the old ones and deleting them, went smoothly.

Two or three days later pods began restarting repeatedly or getting stuck in Terminating, and node CPU and memory load made the nodes unusable. The cause was a bug in the AMI the managed node groups were created with. The old node groups were gone, and managed node groups accept only the latest AMI version, so going back to the previous AMI was refused. After an AMI with the fix was released, the managed node group update to it failed, because the update evicts pods and the pods could not stop.

Enigmo finally ran the managed node group update through CloudFormation while watching `kubectl get pods --all-namespaces`, and whenever a pod stayed in Terminating for a while it killed it with `kubectl delete pods <pod name> --grace-period=0 --force`.

Severity
–
Scope
Pods on the EKS cluster's managed node groups restarted repeatedly or could not stop, and nodes became unusable under CPU and memory load
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

The old node groups had already been deleted, so the pods could not be moved back. While keeping the pods running as best it could, the team found it had hit a known bug, "Pods stuck in terminating state after AMI amazon-eks-node-1.16.15-20201112". An attempt to switch the managed node groups to the older AMI with CloudFormation failed with `Requested Nodegroup release version 1.15.11-20201007 is invalid. Allowed release version is 1.15.12-20201112 (Service: AmazonEKS; Status Code: 400; Error Code: InvalidParameterException`, because managed node groups can only use the latest AMI version.

When an AMI with the fix was released, updating the managed node groups to it also failed. The managed node group update taints the nodes, drains them and evicts the pods, and because pods could not stop, the eviction failed and with it the node group update.

Symptom Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall

Root cause

Root cause Breaking change in an upgrade

Component Node operating system imageNode pool
Contributing factor Ineffective rollbackChange with a delayed effect
Trigger
Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

Detection

–

Remediation

Action items

Lessons

As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Node operating system image, Node pool among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.