Incident Library
Applied filters Root causeBreaking change in an upgradeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases14 cases

13 to 14 of 14

  1. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
    Symptoms

    From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

    DNS resolution failuresImage pull failuresMonitoring outage
    Trigger

    Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

    Root cause
    Breaking change in an upgrade
    Component
    Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache
    Contributing factor
    No staged rolloutExcessive blast radiusAutomation amplification
    Detection

    Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

    Remediation
    • pointed customers to Canonical's description of the systemd bug and its fix
    • recommended rebooting affected VMs or running a template script that repairs DNS resolution
    • built automatic detection and remediation for AKS nodes and rolled it out across all regions
    • mitigated the remaining AKS customers manually through support
    Action items
    • Review the process that lets critical security updates bypass Safe Deployment Practices
    • Improve monitoring of the AKS data plane to alert on an uptick of errors
    • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
    • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
    • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
    • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
    • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
    • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)
    Lessons

    Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

    View case
  2. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021
    Symptoms

    At 13:15 on 9 June 2021 a user reported in the #ask-cloud-platform channel that deploying a UAT application failed on its Ingress resource.

    Deploys failing
    Trigger

    The Cloud Platform team upgraded the ModSec ingress controller module to v3.33.0; the deployment appeared to succeed.

    Root cause
    Breaking change in an upgrade
    Component
    Admission webhookIngress controller
    Contributing factor
    Unrepresentative staging
    Detection

    A user reported the deployment error in the #ask-cloud-platform Slack channel at 13:15.

    Remediation
    • rolled back the ModSec ingress controller module to v0.0.7
    Action items
    • Find out why the issue was not flagged in the test cluster and try to reproduce it, possibly with a new test (ticket #2972)
    • Add a smoke test that checks the alerts in Alertmanager (ticket #2973)
    • Add a helloworld app that uses the ModSec controller so smoke tests can check that traffic works (ticket #2974)
    • Make a new version of the ModSec module work on EKS for both live-1 and live; neither the old nor the new version works on live (ticket #2975)
    View case
Check your cluster for this

These 14 cases involve Ingress controller and Node operating system image. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Breaking change in an upgrade14 cases

Component
Ingress controller5Node operating system image4Admission webhook3Cloud load balancer2Client-side DNS resolver and cache2GKE control plane2Linux kernel2systemd-networkd2Calico1cert-manager1Cilium1Cluster autoscaler1ConfigMaps and Secrets1Application database1etcd1Istio1Kubernetes API server1kube-proxy1kubelet1Node pool1Persistent volumes and stateful sets1Container registry and its credentials1Web application firewall rules1
Contributing factor
No staged rollout5Unrepresentative staging4Automation amplification3Change with a delayed effect3Deferred upgrades3Monitoring blind spot2Single point of failure2Unverified assumption1Untested backup1Excessive blast radius1Configuration not in code1Unreviewed defaults1Unpatched known bug1Limited debug access1No headroom1Ineffective rollback1
Organisation
2i2c2OpenAI2UK Ministry of Justice2Altinity1Datadog1GitHub1Honeycomb1Microsoft Azure1Reddit1Sunwoo Park1Wealthy1
  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created
    2i2c2026
  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates
    2i2c2026
  3. IL-0092 Upgrading ingress-nginx from 1.12.0 to 1.14.3 brought a newer ModSecurity and OWASP Core Rule Set whose false positives blocked HMPPS Auth, so users could not sign in to their applications
    UK Ministry of Justice2026
  4. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  5. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
  6. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  7. IL-0201 A framework upgrade caused kernel panics in GitHub's Kubernetes infrastructure, and GitHub.com web and API requests, mostly for issues and pull requests, failed intermittently for about 16 hours
    GitHub2025
  8. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
  9. IL-0203 A routine Kubernetes cluster system update made the AWS Load Balancer Controller fail, so new pods were never registered with the ALB and Honeycomb's EU1 ingest and UI went down once the last old pods were replaced
    Honeycomb2024
  10. IL-0180 A GKE control plane auto-upgrade from 1.22 to 1.23 removed the v1beta1 Ingress and ValidatingWebhookConfiguration APIs that Istio 1.7.4 still used, and Istio stopped working in production
    Sunwoo Park2023
  11. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023
  12. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  13. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022
  14. IL-0090 After an upgrade of the ModSec ingress controller module, its validating webhook served a certificate for the wrong service name and blocked every Ingress create and change
    UK Ministry of Justice2021