Cases1 case

1 to 1 of 1

1. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False

   Inflab2025

   Symptoms

   GPU nodes for an AI video upscaling job were created normally and reached Ready, and then about 10 to 11 minutes later Karpenter treated each one as unhealthy and replaced it. Every new node went the same way, so the GPU pods kept restarting and the upscaling work could not make progress.

   Node recreation loopGPU unavailable

   Trigger

   Enabling Karpenter's node auto-repair feature gate (nodeRepair: true) and then running GPU pods on nodes from the Bottlerocket NVIDIA AMI.

   Root cause

   Software defect

   Component

   KarpenterNode operating system imageGPU driver

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   The team noticed GPU nodes being replaced repeatedly after running GPU pods, then found the DCGMError message in the node events.

   Remediation

   • Disabled Karpenter node auto-repair by setting the nodeRepair feature gate to false

   Lessons

   Automatic node repair acts on every unhealthy node condition, including AcceleratedHardwareReady, so an image defect that only breaks GPU health monitoring is enough to have every GPU node terminated and recreated in a loop. Before turning on auto-repair, check which conditions it watches and whether the node image reports them correctly. Disabling auto-repair stops the loop at once but is only a temporary measure; installing the missing library or changing the AMI address the cause but cost more on an immutable OS such as Bottlerocket.

   View case

By root cause1