IL-0029 Buffer · 2020-08-31 (published)kubernetesaws
Kubernetes enforces a container's CPU limit with the kernel's CFS quota: in each period (100ms by default) a container may use only its share of CPU time and is throttled after that. Buffer had set CPU limits on its services because that was the common recommendation, and because without limits a busy container had earlier starved the kubelet and turned nodes NotReady. The team then saw throttling on containers whose usage was far below the limit, and traced it to a Linux kernel bug that throttled containers with CPU limits unnecessarily. The fix for that bug is in kernel 4.19 and later, but the kops version Buffer ran still used an older host image, so upgrading the kernel was not yet possible.
Instead, after long discussion, the team removed CPU limits from every service that sat directly or indirectly on the users' critical path, and did it with safeguards. Those services were moved to dedicated tainted nodes, with tolerations, so that an unbounded service could only disturb other unbounded ones. Each service's CPU request was set from a few months of Datadog usage data: the peak usage plus a margin of more than 20%, with memory requests set the same way, so the scheduler reserved enough room on the node. The cost was lower container density and idle capacity at quiet hours.
After a few weeks, latency improved across all the modified services; the main landing page buffer.com became up to 22 times faster. The author adds that upgrading to a fixed kernel should be preferred over removing limits, and that he could not yet confirm the kernel fix removes throttling completely.
Services on Buffer's kops-managed cluster on AWS (about 60 nodes and 1500 containers) answered more slowly than they should have. The container throttling metric showed that many containers were being throttled whatever their CPU usage was, including containers that never came close to their CPU limit.
One of the main APIs is the example in the write-up: its CPU limit was 800m, its peak usage was at most 200m, and the throttling graph still showed throttling. On paper the service had four times the CPU it needed, yet the kernel was holding it back.
The opposite setting had already hurt the team before. When containers ran without CPU limits and used up a node's CPU, the kubelet on that node stopped responding, the node went NotReady, and the pods that were moved elsewhere carried the same problem onto the next nodes. That history is why removing the limits was a long discussion and not a quick fix.
Root cause Software defect
While reviewing the container throttling metric, the team noticed many containers being throttled even though their CPU usage stayed far below their limits.
A CPU limit can slow a service down even when the service uses a small part of it, because of how the kernel enforces CFS quota on older kernels; the throttling metric shows it, the CPU usage graph does not. Upgrading the kernel is the safer fix. Removing limits works for latency, but only together with honest CPU and memory requests, isolation of the unbounded services, and monitoring of node usage, since unbounded containers can starve the kubelet.
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared Linux kernel, Node operating system image
IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeoutsOmio · 2020same root cause · shared Linux kernel, Container CPU limit
IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019same root cause · shared Linux kernel, Node operating system image
IL-0171 Karpenter node auto-repair replaced EKS GPU nodes about 10 minutes after they became Ready, in a loop, because the Bottlerocket NVIDIA AMI lacked libdcgm.so and set AcceleratedHardwareReady to FalseInflab · 2025same root cause · shared Node operating system image
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node operating system image
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025same root cause · shared Linux kernel
IL-0175 Lowering a JVM pod's CPU request from 2 to 1 pushed memory past the 90% alert because the JVM treats a cgroup v1 cpu.shares value of exactly 1024 as unset and sized Reactor Netty threads from the node's 36 CPUsHyperconnect · 2022same root cause · shared Container CPU limit
IL-0146 New AKS clusters with Azure CNI in Central India lost tunnel connectivity because VMs began getting MAC addresses with a 06 prefix, which exposed a dormant bug in bridge-mode Azure CNIMicrosoft Azure · 2020same root cause · shared Node operating system image