Cases19 cases

9 to 13 of 19

1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts

   DFDS2022

   Symptoms

   On 2022-10-31 a planned upgrade of the vpc-cni EKS add-on, started from the deployment pipeline at 13:45 UTC, timed out at 14:10. Only about 10 of the 45 aws-node pods had upgraded, 5 were stuck in Terminating and the other 30 had not been attempted.

   Node NotReadyUpgrade stallIntermittent timeoutsIntermittent connection failures

   Trigger

   At 07:30 UTC on 2022-10-31 the daily node rollover replaced the existing nodes with nodes built from AMI v20221027, the latest AWS EKS node AMI, released on 2022-10-29 and later recalled because of a kernel issue.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernelkubeletAWS VPC CNINode launch templateTerraform

   Contributing factor

   No staged rolloutAutomation amplificationOverlapping failuresMonitoring blind spotNo health probes

   Detection

   The team noticed nodes flapping between NodeNotReady and NodeReady ("KubeletNotReady PLEG is not healthy") at 14:30 UTC while watching the stalled vpc-cni upgrade, about seven hours after the rollover. Time to detection was 7 hours. Alerting did not catch the NodeNotReady events or the socket growth.

   Remediation

   • cordoned, drained and deleted 15 stuck nodes one at a time to complete the vpc-cni upgrade
   • changed the Terraform infrastructure-modules so a specific AMI could be set
   • rolled the EC2 Auto Scaling group launch configuration back to the previous known good AMI through the pipeline

   Action items

   • Reach out to teams with suggestions about readiness and liveness probes
   • Create a dashboard for readiness and liveness probe coverage
   • Write a wiki article on readiness and liveness probes and their usefulness
   • Modify infrastructure-modules to allow specifying a particular AMI
   • Schedule a discussion on whether to take the latest AMI immediately and how to test new AMIs before they are deployed
   • Implement alerting on abnormal NodeNotReady events and on increased socket use
   • Create a learning path for Kubernetes within the team
   • Collect logs from nodes centrally
   • Create a Prometheus exporter that retrieves the latest available AMI, with a dashboard and alerts

   Lessons

   When a new node AMI has just been rolled out, upgrading other cluster components on the same day makes it hard to tell which change is causing trouble; here the vpc-cni upgrade on the same day confused matters. Taking the latest AMI automatically with no way to pin one meant a rollback first needed a Terraform change. The team also concluded that it needs better communication paths with application teams during incidents.

   View case
2. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff

   Microsoft Azure2022

   Symptoms

   From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

   DNS resolution failuresImage pull failuresMonitoring outage

   Trigger

   Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache

   Contributing factor

   No staged rolloutExcessive blast radiusAutomation amplification

   Detection

   Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

   Remediation

   • pointed customers to Canonical's description of the systemd bug and its fix
   • recommended rebooting affected VMs or running a template script that repairs DNS resolution
   • built automatic detection and remediation for AKS nodes and rolled it out across all regions
   • mitigated the remaining AKS customers manually through support

   Action items

   • Review the process that lets critical security updates bypass Safe Deployment Practices
   • Improve monitoring of the AKS data plane to alert on an uptick of errors
   • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
   • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
   • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
   • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
   • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
   • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)

   Lessons

   Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

   View case
3. IL-0501 Every node of the UToronto Kubernetes cluster on Azure went NotReady and the JupyterHub became unreachable

   2i2c2022

   Symptoms

   On 30 August 2022 at 11:15 AM Pacific time, the UToronto JupyterHub at jupyter.utoronto.ca stopped responding and users could not reach it at all. The outage was reported through 2i2c's support desk.

   Service outageNode NotReady

   Trigger

   Microsoft Azure rolled out an Ubuntu update to all Ubuntu 18.04 nodes, which triggered a systemd bug that broke networking.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system image

   Contributing factor

   Limited debug access

   Detection

   The outage was reported at 11:15 AM through 2i2c's Freshdesk support desk because jupyter.utoronto.ca was not responding. At 11:25 AM the customer's representative forwarded the Microsoft advisory on the same ticket, which pointed the team to the cause.

   Remediation

   • Started an upgrade of the cluster's Kubernetes and node versions, which replaced all old nodes with new working ones

   Action items

   • Document how to SSH into the Kubernetes nodes
   • Find a way to be notified of Azure advisories

   Lessons

   Once the cause was known, a single action brought the hub back quickly. The team had no documented way to SSH into its Kubernetes nodes, and the customer's representative saw the Azure advisory and pointed it out on the support ticket, which saved a lot of time.

   View case
4. IL-0462 Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacement

   Shopify2021

   Symptoms

   In 2021 Shopify's KateSQL, a database-as-a-service platform that runs several hundred production MySQL instances on GKE, found that some MySQL pods started and shut down much more slowly than similar pods with the same data set. Instance replacement, the procedure that creates a new MySQL replica pod and stops the old one, showed pod creation times ranging from 10 to 30 minutes, which held up maintenance such as configuration changes and upgrades.

   Slow pod startup or shutdownHigh latency

   Root cause

   Software defect

   Component

   Linux kernelNode operating system imageDatabase

   Contributing factor

   Unpatched known bug

   Detection

   KateSQL's instance replacement operations showed inconsistent MySQL pod creation times, and the team then measured the mysql-init container and MySQL startup times across pods.

   Remediation

   • Cordoned older nodes, moved MySQL pods to new nodes through instance replacement and drained the old nodes
   • Forced the kubepods memory cgroup to empty its dead cgroups on an affected node to test the fix
   • Created new GKE 1.18 clusters and started creating MySQL pods on them

   Lessons

   The investigation was one of the longest the Database Platform group had carried out, because of the nature of the problem, how hard it was to reproduce and the lack of an established method to follow; once the cause was known, there were several ways to fix it.

   View case
5. IL-0449 Pods on Enigmo's new EKS managed node groups kept restarting or stuck Terminating and the nodes became unusable

   Enigmo2020

   Symptoms

   Two or three days after Enigmo created EKS managed node groups and moved its pods there from the old node groups, the cluster suddenly became unstable: pods restarted again and again or could not stop and stayed around indefinitely, node CPU and memory load rose high, and the nodes became unusable from the cluster.

   Pods stuck in TerminatingSustained high node CPUMemory pressureNode NotReadyUpgrade stall

   Trigger

   Enigmo created EKS managed node groups that used an AMI containing the bug and moved its pods onto them, as preparation for upgrading EKS from 1.15 to 1.16.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imageNode pool

   Contributing factor

   Ineffective rollbackChange with a delayed effect

   Remediation

   • Changed the managed node groups' AMI version to the fixed release
   • Ran the managed node group update through CloudFormation
   • Watched the pods across all namespaces during the update
   • Force-deleted every pod that stayed in Terminating with kubectl delete --grace-period=0 --force

   Action items

   • Check the latest releases of the cluster's modules quarterly or monthly and upgrade them regularly

   Lessons

   As long as a team uses Amazon EKS or Kubernetes, it has to keep track of upgrades of the cluster and of modules such as ingress controllers, because a release with breaking changes can be announced without warning after a long stable period. Checking releases and upgrading on a regular schedule keeps the effect of each upgrade on the service small.

   View case

By root cause6