Cases2 cases

1 to 2 of 2

1. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries

   Microsoft Azure2025

   Symptoms

   Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.

   Scale-up failureUpgrade stall

   Trigger

   A node image auto-upgrade from 202507.* to 202508.* during planned maintenance, on clusters whose Kubernetes patch version was not pre-cached in the new image.

   Root cause

   Software defect

   Component

   Node operating system imageNode pool

   Contributing factor

   Cache-masked failureAutomation amplification

   Detection

   The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.

   Remediation

   • The customer pinned the node image to a 202507.* snapshot on clusters running 1.31.8
   • The customer upgraded clusters to AKS 1.31.9 or 1.31.10, whose binaries are cached in the 202508.* images
   • The AKS team hot-patched VHD 20250811 so the proxy configuration is used for uncached binary downloads and made it globally available in public regions on 2025-09-04; it also announced the hotfix for VHD 20250820
   • The AKS team told users to reimage their nodes with az aks nodepool upgrade --node-image-only to pick up the hotfixed image

   Action items

   • Apply the proxy fix in all future VHDs

   Lessons

   A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path. For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.

   View case
2. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff

   Microsoft Azure2022

   Symptoms

   From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

   DNS resolution failuresImage pull failuresMonitoring outage

   Trigger

   Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache

   Contributing factor

   No staged rolloutExcessive blast radiusAutomation amplification

   Detection

   Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

   Remediation

   • pointed customers to Canonical's description of the systemd bug and its fix
   • recommended rebooting affected VMs or running a template script that repairs DNS resolution
   • built automatic detection and remediation for AKS nodes and rolled it out across all regions
   • mitigated the remaining AKS customers manually through support

   Action items

   • Review the process that lets critical security updates bypass Safe Deployment Practices
   • Improve monitoring of the AKS data plane to alert on an uptick of errors
   • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
   • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
   • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
   • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
   • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
   • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)

   Lessons

   Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

   View case

By root cause2