Cases16 cases

11 to 15 of 16

1. IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNI

   Microsoft Azure2020

   Symptoms

   In newly created AKS clusters using Azure CNI in Central India, kubectl exec and kubectl logs could time out, and pod-to-pod or pod-to-node traffic could time out. The cluster diagnostics page showed tunnel exec issues. Clusters using kubenet were not affected.

   Intermittent timeouts

   Trigger

   VMs in Central India started being assigned MAC addresses in a much higher range than before (06 prefix rather than the usual 00).

   Root cause

   Software defect

   Component

   Azure CNINode operating system image

   Contributing factor

   –

   Remediation

   • told owners of affected scale-set clusters to run a node image upgrade, or any Kubernetes upgrade such as a patch version, to get the Azure CNI that does not use a bridge
   • told owners of availability-set clusters to scale out and delete the VMs with a high-range MAC address in bridge mode
   • advised recreating affected clusters that were brand new

   Action items

   • Roll out a hotfix that makes transparent mode the default in all regions

   Lessons

   A change in how the platform assigns MAC addresses can wake a long-dormant bug in the network plugin. Knowing which CNI mode the nodes run (bridge or transparent) and keeping node images current decides whether such a bug reaches you.

   View case
2. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit

   Buffer2020

   Symptoms

   Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.

   CPU throttlingHigh latencyNode NotReady

   Trigger

   CPU limits on user-facing services, enforced through CFS quota by a Linux kernel with a bug that throttled containers even when they were well under their limit.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelNode operating system image

   Contributing factor

   Unpatched known bugUnverified assumption

   Detection

   While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.

   Remediation

   • removed CPU limits from all services directly or indirectly on the users' critical path
   • moved those services to dedicated tainted nodes and gave them matching tolerations
   • set each service's CPU request to its peak usage over a few months plus a margin of more than 20%, and set memory requests the same way

   Action items

   • Move nodes to a kernel that contains the CFS throttling fix (4.19 or later), which for Buffer means a kops version whose host image is Ubuntu 20.04
   • After the kernel upgrade, check whether throttling is fully gone and update the post

   Lessons

   A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.

   View case
3. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image

   Google Cloud2019

   Symptoms

   Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.

   Node kernel panic

   Trigger

   Upgrading GKE node pools to one of the affected 1.13 or 1.14 versions.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernel

   Contributing factor

   –

   Remediation

   • advised affected users to downgrade to a patch release not on the affected list, with support helping Release Channel users
   • released GKE 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12, which downgraded Container-Optimized OS to cos-73-11647-293-0
   • rolled the fixed versions out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves

   Action items

   • Ship a GKE release with an upgraded kernel that fixes the panics

   Lessons

   Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.

   View case
4. IL-0057 Higher p99 latency after moving a reverse proxy to EKS, from outgoing connections delayed by a SNAT conntrack race

   Prezi2019

   Symptoms

   Prometheus alerts fired on tail latency: from 7/24, the p99 response time of a few services went up after Prezi moved its Nginx-based reverse proxy, smartgateway, from dedicated Elastic Beanstalk machines onto shared Kubernetes worker nodes. The bump affected only the tail, so the earlier monitoring of average response times would most certainly have missed it.

   High latencyIntermittent timeouts

   Trigger

   Moving the reverse proxy onto shared EKS worker nodes, where its outgoing connections went through source NAT on the node and some SYN packets were dropped by a race in Netfilter's conntrack.

   Root cause

   DNS or network fault

   Component

   conntrack tableAWS VPC CNILinux kernelNode operating system image

   Contributing factor

   Unreviewed defaultsUnpatched known bug

   Detection

   Prometheus alerts on p99 latency fired after the reverse proxy migration.

   Remediation

   • added AWS X-Ray tracing and an $upstream_connect_time trace attribute to locate the delay
   • ran more pods of the service on more nodes, which reduced but did not end the delays
   • set AWS_VPC_K8S_CNI_EXTERNALSNAT=true on the AWS VPC CNI to turn off SNAT on the nodes

   Lessons

   Alerting on tail latency instead of averages caught a problem that would otherwise have gone unnoticed, and tracing turned a vague slowdown into a measured connect delay. Delays of exactly 1 or 3 seconds on new outgoing connections point to dropped SYN packets, and on EKS the SNAT done by the VPC CNI is where to look.

   View case
5. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy

   Microsoft Azure2019

   Symptoms

   From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

   Upgrade stall

   Trigger

   A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

   Root cause

   Faulty deployment

   Component

   Node operating system imageNode pool

   Contributing factor

   –

   Detection

   Customers received failure messages when they tried to upgrade their clusters.

   Remediation

   • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
   • mitigated affected clusters by hand (AKS and VMSS teams)
   • rolled out hot fixes to all regions, completed by 15 October 2019

   View case

By root cause6