Cases6 cases

3 to 6 of 6

1. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False

   Inflab2025

   Symptoms

   GPU nodes for an AI video upscaling job were created normally and reached Ready, and then about 10 to 11 minutes later Karpenter treated each one as unhealthy and replaced it. Every new node went the same way, so the GPU pods kept restarting and the upscaling work could not make progress.

   Node recreation loopGPU unavailable

   Trigger

   Enabling Karpenter's node auto-repair feature gate (nodeRepair: true) and then running GPU pods on nodes from the Bottlerocket NVIDIA AMI.

   Root cause

   Software defect

   Component

   KarpenterNode operating system imageGPU driver

   Contributing factor

   Automation amplificationUnverified assumption

   Detection

   The team noticed GPU nodes being replaced repeatedly after running GPU pods, then found the DCGMError message in the node events.

   Remediation

   • Disabled Karpenter node auto-repair by setting the nodeRepair feature gate to false

   Lessons

   Automatic node repair acts on every unhealthy node condition, including AcceleratedHardwareReady, so an image defect that only breaks GPU health monitoring is enough to have every GPU node terminated and recreated in a loop. Before turning on auto-repair, check which conditions it watches and whether the node image reports them correctly. Disabling auto-repair stops the loop at once but is only a temporary measure; installing the missing library or changing the AMI address the cause but cost more on an immutable OS such as Bottlerocket.

   View case
2. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries

   Microsoft2025

   Symptoms

   Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.

   Scale-up failureUpgrade stall

   Trigger

   A node image auto-upgrade from 202507.* to 202508.* during planned maintenance, on clusters whose Kubernetes patch version was not pre-cached in the new image.

   Root cause

   Software defect

   Component

   Node operating system imageNode pool

   Contributing factor

   Cache-masked failureAutomation amplification

   Detection

   The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.

   Remediation

   • The customer pinned the node image to a 202507.* snapshot on clusters running 1.31.8
   • The customer upgraded clusters to AKS 1.31.9 or 1.31.10, whose binaries are cached in the 202508.* images
   • The AKS team hot-patched VHD 20250811 so the proxy configuration is used for uncached binary downloads and made it globally available in public regions on 2025-09-04; it also announced the hotfix for VHD 20250820
   • The AKS team told users to reimage their nodes with az aks nodepool upgrade --node-image-only to pick up the hotfixed image

   Action items

   • Apply the proxy fix in all future VHDs

   Lessons

   A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path. For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.

   View case
3. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
4. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes

   Google Cloud2022

   Symptoms

   From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

   High error rateResources deletedService outage

   Trigger

   A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

   Root cause

   Software defect

   Component

   Model request routing

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

   Remediation

   • restored the Vertex Prediction Endpoints that had been marked as deleted

   View case

By root cause1