Incident Library
Applied filters SymptomScale-up failureFactorCache-masked failureOrganisationMicrosoftClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft2025
    Symptoms

    Creating or scaling a node pool on AKS 1.31.8 failed: the VMSS instances never finished provisioning. This started after planned maintenance moved the node image from 202507.* to 202508.*, in a cluster where nodes must send all outbound traffic through an internal HTTP/HTTPS proxy and direct egress is blocked. It happened with both Ubuntu and AzureLinux images.

    Scale-up failureUpgrade stall
    Trigger

    A node image auto-upgrade from 202507.* to 202508.* during planned maintenance, on clusters whose Kubernetes patch version was not pre-cached in the new image.

    Root cause
    Software defect
    Component
    Node operating system imageNode pool
    Contributing factor
    Cache-masked failureAutomation amplification
    Detection

    The customer saw VMSS instance provisioning fail with a vmssCSE error in the VMSS activity logs and reported it as a GitHub issue.

    Remediation
    • The customer pinned the node image to a 202507.* snapshot on clusters running 1.31.8
    • The customer upgraded clusters to AKS 1.31.9 or 1.31.10, whose binaries are cached in the 202508.* images
    • The AKS team hot-patched VHD 20250811 so the proxy configuration is used for uncached binary downloads and made it globally available in public regions on 2025-09-04; it also announced the hotfix for VHD 20250820
    • The AKS team told users to reimage their nodes with az aks nodepool upgrade --node-image-only to pick up the hotfixed image
    Action items
    • Apply the proxy fix in all future VHDs
    Lessons

    A bootstrap path that runs only when an artefact is missing from a cache is easy to break without noticing: here only clusters on Kubernetes versions not pre-cached in the node image ever downloaded binaries, so only they met the proxy regression. Testing node provisioning behind a mandatory egress proxy, on older patch versions as well as the newest, would exercise that path. For clusters with restricted egress, pinning the node image or moving to a Kubernetes version cached in the image were both quick workarounds while the fix was shipped.

    View case
Check your cluster for this

This case involves Node operating system image and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Node operating system imageNode pool
Contributing factor
Automation amplificationCache-masked failure
Organisation
Microsoft
  1. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft2025