Cases6 cases

5 to 6 of 6

1. IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI

   Microsoft Azure2020

   Symptoms

   In newly created AKS clusters using Azure CNI in Central India, kubectl exec and kubectl logs could time out, and pod-to-pod or pod-to-node traffic could time out. The cluster diagnostics page showed tunnel exec issues. Clusters using kubenet were not affected.

   Intermittent timeouts

   Trigger

   VMs in Central India started being assigned MAC addresses in a much higher range than before (06 prefix rather than the usual 00).

   Root cause

   Software defect

   Component

   Azure CNINode operating system image

   Contributing factor

   –

   Remediation

   • told owners of affected scale-set clusters to run a node image upgrade, or any Kubernetes upgrade such as a patch version, to get the Azure CNI that does not use a bridge
   • told owners of availability-set clusters to scale out and delete the VMs with a high-range MAC address in bridge mode
   • advised recreating affected clusters that were brand new

   Action items

   • Roll out a hotfix that makes transparent mode the default in all regions

   Lessons

   A change in how the platform assigns MAC addresses can wake a long-dormant bug in the network plugin. Knowing which CNI mode the nodes run (bridge or transparent) and keeping node images current decides whether such a bug reaches you.

   View case
2. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy

   Microsoft Azure2019

   Symptoms

   From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

   Upgrade stall

   Trigger

   A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

   Root cause

   Faulty deployment

   Component

   Node operating system imageNode pool

   Contributing factor

   –

   Detection

   Customers received failure messages when they tried to upgrade their clusters.

   Remediation

   • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
   • mitigated affected clusters by hand (AKS and VMSS teams)
   • rolled out hot fixes to all regions, completed by 15 October 2019

   View case

By root cause4