IL-0144 Microsoft Azure · 2022-08-30kubernetesazurenetworking
At 06:00 UTC on 30 August 2022 Canonical published an Ubuntu security update. Azure VMs on Ubuntu 18.04 with unattended-upgrades enabled, which by default downloads and applies security updates once a day, installed it, including systemd 237-3ubuntu10.54. A race-condition bug in that systemd version (Launchpad bug 1988119), set off by this update combined with an earlier one, made the VMs lose their DNS configuration. The bug only affects systems whose network configuration identifies the network interface card by driver name, which is why it hit Azure and not other major cloud providers.
Because of their criticality, security updates like this one do not go through Azure's Safe Deployment Practices process; Azure says it is reviewing that process. AKS nodes on Ubuntu 18.04 lost DNS, kubelet could not resolve registry hostnames, and new pods could not pull their images. Downstream services built on these VMs, including Azure Monitor, Application Insights, Log Analytics and Microsoft Sentinel, were affected by the same DNS failures.
Mitigation was done partly together with Canonical. Azure pointed customers to Canonical's description of the bug and a fix, and recommended rebooting affected VMs or, where that was not possible, running a template script to repair DNS after testing it. For AKS nodes, the AKS team built automatic detection and remediation and rolled it out in all regions, which resolved the vast majority of customers; the rest needed manual mitigation through support. Impact lasted until 16:00 UTC on 31 August 2022.
From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.
On AKS, kubelet could not resolve the DNS names of container registries, so pod creation failed with errors such as ImagePullBackoff. Other Azure services that run on Ubuntu 18.04 VMs failed the same way: customers of Azure Monitor, Application Insights, Log Analytics and Microsoft Sentinel may have been unable to reach their log data and may have seen Log Search alerts and Microsoft Sentinel alerts missed or delayed.
The problem was limited to Ubuntu 18.04 but appeared in every Azure region, public and sovereign clouds alike. Incoming customer support cases confirmed that only those Ubuntu versions were affected.
Symptom DNS resolution failuresImage pull failuresMonitoring outage
Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.
Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.
IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hoursWealthy · 2025 · sev1same root cause · shared Client-side DNS resolver and cache
IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the APIOpenAI · 2025same root cause · shared Node operating system image
IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress trafficAltinity · 2025same root cause · shared Node operating system image
IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providersDatadog · 2023same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.