Incident Library
Applied filters ComponentNode operating system imageFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
    Symptoms

    From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

    DNS resolution failuresImage pull failuresMonitoring outage
    Trigger

    Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache
    Contributing factor
    No staged rolloutExcessive blast radiusAutomation amplification
    Detection

    Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

    Remediation
    • pointed customers to Canonical's description of the systemd bug and its fix
    • recommended rebooting affected VMs or running a template script that repairs DNS resolution
    • built automatic detection and remediation for AKS nodes and rolled it out across all regions
    • mitigated the remaining AKS customers manually through support
    Action items
    • Review the process that lets critical security updates bypass Safe Deployment Practices
    • Improve monitoring of the AKS data plane to alert on an uptick of errors
    • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
    • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
    • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
    • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
    • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
    • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)
    Lessons

    Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

    View case
Check your cluster for this

These 6 cases involve Node operating system image and GPU driver. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause3

Software defect3 cases

Component
Node operating system image3GPU driver1Karpenter1kubelet1Node launch template1Linux kernel1Node pool1Terraform1AWS VPC CNI1
Contributing factor
Automation amplification3Unverified assumption1Cache-masked failure1Monitoring blind spot1No health probes1No staged rollout1Overlapping failures1
Organisation
DFDS1Inflab1Microsoft1
  1. IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to False
    Inflab2025
  2. IL-0147 AKS nodes behind an HTTP proxy failed to provision with CSE exit code 31 on 202508 node images, because a refactoring stopped the proxy from being used when downloading uncached Kubernetes binaries
    Microsoft2025
  3. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022

Dependency failure1 case

Component
GPU driverNode operating system imageNode pool
Contributing factor
Automation amplification
Organisation
Google Cloud
  1. IL-0021 GKE GPU nodes on Container-Optimized OS could not download NVIDIA drivers (403 Forbidden) and came up without working GPUs
    Google Cloud2024

Breaking change in an upgrade2 cases

Component
Node operating system image2Cilium1Client-side DNS resolver and cache1kubelet1Container registry and its credentials1systemd-networkd1
Contributing factor
Automation amplification2No staged rollout2Excessive blast radius1Unreviewed defaults1Change with a delayed effect1
Organisation
Datadog1Microsoft Azure1
  1. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  2. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022