IL-0146 Microsoft Azure · 2020-12-04kubernetesazurenetworking
From about 00:00 UTC on 4 December 2020 to 00:10 UTC on 15 December 2020, newly created AKS clusters with Azure CNI in Central India could have tunnel connectivity problems. VMs in the region began to receive MAC addresses with a 06 prefix instead of the usual 00, and that revealed a bug in Azure CNI that had been dormant for a long time. The newest Azure CNI, which no longer uses a bridge, was not affected.
The AKS team published a way to check each node with az vmss run-command (or az vm run-command for availability sets) for the MAC address and the CNI mode. The fix was to get the newer CNI: on scale sets a node image upgrade, or any Kubernetes upgrade such as a patch version; on availability sets, scaling out and deleting the affected VMs; for a brand-new cluster, recreating it. A hotfix to make transparent mode the default in all regions was in progress; existing clusters still had to be upgraded.
In newly created AKS clusters using Azure CNI in Central India, kubectl exec and kubectl logs could time out, and pod-to-pod or pod-to-node traffic could time out. The cluster diagnostics page showed tunnel exec issues. Clusters using kubenet were not affected.
On an affected node, ip addr show eth0 showed a MAC address beginning link/ether 60:45:.. and /etc/cni/net.d/10-azure.conflist showed "mode":"bridge". A node with link/ether 00:0d:.. in bridge mode, or 60:45:.. in "mode":"transparent", was not affected.
Symptom Intermittent timeouts
Root cause Software defect
–
A change in how the platform assigns MAC addresses can wake a long-dormant bug in the network plugin. Knowing which CNI mode the nodes run (bridge or transparent) and keeping node images current decides whether such a bug reaches you.
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025same root cause · shared Node operating system image
IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binariesMicrosoft · 2025same root cause · shared Node operating system image
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared Node operating system image
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node operating system image