IL-0147 Microsoft · 2025-08-26kubernetesazurenetworking
AKS node images (VHDs) pre-cache the Kubernetes node binaries for the two most recent patch versions of each minor release. When a node boots on a version that is not cached, the Custom Script Extension (CSE) bootstrap downloads the binaries from acs-mirror.azureedge.net. A recent refactoring in the bootstrap stopped the HTTP proxy configuration from being used for that download, so in clusters where nodes can reach the internet only through a proxy, the download tried direct egress, timed out, and the CSE exited with code 31 once it passed its 780-second limit. Node provisioning failed.
The affected node image versions were 20250806, 20250811 and 20250820. Clusters on a version whose binaries were cached in the image, such as v1.31.9 and v1.31.10 in 20250806 and 20250811, were not affected, which is why the bug showed only on AKS 1.31.0 to 1.31.8 and other uncached versions. The issue was opened on 2025-08-27; the AKS team confirmed the cause on 2025-09-03, and VHD 20250811 with a hotfix was globally available in public regions on 2025-09-04.
Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.
The VMSS activity log showed "VM has reported a failure when processing extension 'vmssCSE'" with "command terminated with exit status=31". The CSE output showed curl repeatedly trying to download https://acs-mirror.azureedge.net/kubernetes/v1.31.8/binaries/kubernetes-node-linux-amd64.tar.gz by connecting straight to the IP addresses of acs-mirror.azureedge.net on port 443, each attempt ending in a 60-second timeout, until "Error: CSE has been running for 806 seconds, exceeding the limit of 780 seconds." and "CSE timeout approaching, exiting early."
The reporter found that 202507.* images on the same AKS versions worked, and that 202508.* images worked on AKS 1.31.9 and 1.31.10. Another user reported that a cluster on AKS 1.32.4 with node image 202507.21.0 worked, but after a reconcile AKS tried 202508.06.0, provisioning failed and the cluster went into the Failed state.
Symptom Scale-up failureUpgrade stall
Root cause Software defect
The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.
A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path.
For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025same root cause · shared Node operating system image
IL-0078 Enabling EKS secrets encryption with KMS through Terraform removed worker node IAM permissions and took down all ZenML Pro workspacesZenML · 2025same root cause · shared Node pool
IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodesLinode · 2024same root cause · shared Node pool
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node operating system image
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020same root cause · shared Node operating system image
IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutesGrafana Labs · 2020same root cause · shared Node pool
IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019same root cause · shared Node operating system image