Cases6 cases

5 to 6 of 6

1. IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limit

   Buffer2020

   Symptoms

   Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.

   CPU throttlingHigh latencyNode NotReady

   Trigger

   CPU limits on user-facing services, enforced through CFS quota by a Linux kernel with a bug that throttled containers even when they were well under their limit.

   Root cause

   Software defect

   Component

   Container CPU limitLinux kernelNode operating system image

   Contributing factor

   Unpatched known bugUnverified assumption

   Detection

   While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.

   Remediation

   • removed CPU limits from all services directly or indirectly on the users' critical path
   • moved those services to dedicated tainted nodes and gave them matching tolerations
   • set each service's CPU request to its peak usage over a few months plus a margin of more than 20%, and set memory requests the same way

   Action items

   • Move nodes to a kernel that contains the CFS throttling fix (4.19 or later), which for Buffer means a kops version whose host image is Ubuntu 20.04
   • After the kernel upgrade, check whether throttling is fully gone and update the post

   Lessons

   A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.

   View case
2. IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS image

   Google Cloud2019

   Symptoms

   Some nodes in Google Kubernetes Engine clusters that had recently been upgraded showed an elevated number of kernel panics. The status page described it as kernel panics following an upgrade in GKE version.

   Node kernel panic

   Trigger

   Upgrading GKE node pools to one of the affected 1.13 or 1.14 versions.

   Root cause

   Software defect

   Component

   Node operating system imageLinux kernel

   Contributing factor

   –

   Remediation

   • advised affected users to downgrade to a patch release not on the affected list, with support helping Release Channel users
   • released GKE 1.13.11-gke.14, 1.13.12-gke.8, 1.14.7-gke.23 and 1.14.8-gke.12, which downgraded Container-Optimized OS to cos-73-11647-293-0
   • rolled the fixed versions out to node pools with auto-upgrade enabled; clusters on manual upgrades had to upgrade their nodes themselves

   Action items

   • Ship a GKE release with an upgraded kernel that fixes the panics

   Lessons

   Downgrading the node operating system image was used as a temporary mitigation until a kernel fix was available.

   View case

By root cause1