Cases2 cases

1 to 2 of 2

1. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes

   Reddit2023

   Symptoms

   About two minutes after an engineer started the Kubernetes 1.23 to 1.24 upgrade of the cluster running Reddit's legacy stack, just after 19:00 UTC, the site stopped. The CDN edge request graph, which is kept separate from the cluster, showed traffic by status collapsing. Users saw the error page, a homepage with no posts, or plain errors.

   Service outageHigh error rateDNS resolution failuresMonitoring outagePods stuck in ContainerCreatingControl plane lag

   Trigger

   An in-place upgrade of the legacy cluster from Kubernetes 1.23 to 1.24; in 1.24 the node-role.kubernetes.io/master label is removed even from running clusters.

   Root cause

   Breaking change in an upgrade

   Component

   CalicoKubernetes API serverAdmission webhooketcd

   Contributing factor

   Configuration not in codeUnrepresentative stagingUntested backupMonitoring blind spotSingle point of failure

   Detection

   The site broke about two minutes into the upgrade; an incident was opened immediately and responders were on the call within three minutes, working from CDN edge graphs because the cluster's own metrics were gone.

   Remediation

   • deleted the Open Policy Agent webhook configurations to stop API server timeouts
   • terminated all worker nodes of the cluster
   • restored the control plane from the pre-upgrade backup onto the first control plane node
   • switched off the cluster autoscaler until the control plane was stable
   • generated new TLS certificates so replacement control plane nodes could join etcd
   • brought traffic back in steps from 1% to 100%
   • restarted legacy services that had been stopped by hand

   Action items

   • Codify cluster configuration so every choice has a record, and document in detail what cannot be codified
   • Standardise components across clusters and replace bespoke configurations
   • Build tooling to manage the lifecycle of whole clusters so they can be recreated or replicated
   • Add pre-production clusters that can be tested with confidence
   • Improve the backup restore process and make cluster upgrades safe

   Lessons

   Configuration that exists only in a live cluster, edited by hand through a special tool, can be forgotten, as the route reflector setup was, and then break on an upgrade nobody connects it to. A Kubernetes upgrade cannot be reverted without a restore, so the restore procedure has to be kept current and practised. Consistent, codified clusters are what make upgrades testable; bespoke clusters keep producing outages.

   View case
2. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff

   Microsoft Azure2022

   Symptoms

   From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

   DNS resolution failuresImage pull failuresMonitoring outage

   Trigger

   Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache

   Contributing factor

   No staged rolloutExcessive blast radiusAutomation amplification

   Detection

   Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

   Remediation

   • pointed customers to Canonical's description of the systemd bug and its fix
   • recommended rebooting affected VMs or running a template script that repairs DNS resolution
   • built automatic detection and remediation for AKS nodes and rolled it out across all regions
   • mitigated the remaining AKS customers manually through support

   Action items

   • Review the process that lets critical security updates bypass Safe Deployment Practices
   • Improve monitoring of the AKS data plane to alert on an uptick of errors
   • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
   • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
   • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
   • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
   • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
   • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)

   Lessons

   Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

   View case

By root cause1