IL-0012 Omio · 2020-02-17 (published)kubernetesgcp
Kubernetes implements CPU requests and CPU limits in entirely different ways, even though they look alike in the pod spec. A request becomes cpu.shares in the cgroup hierarchy under /sys/fs/cgroup/cpu,cpuacct, where 1024 shares stand for one core and the Linux CFS scheduler splits CPU in proportion to the shares, handing unused CPU back to a global pool. A limit becomes a CFS quota: cfs_period_us, always 100000 microseconds (100 ms) unless an alpha, feature-gated option changes it, and cfs_quota_us, the CPU time the container may use in each period. A limit of 2 cores becomes 200 ms per period. Omio's most common machine type has 16 cores. In the write-up's example, a container running ten parallel threads while the other pods are idle uses those 200 ms in 20 ms of wall time, and all its threads are then throttled for the remaining 80 ms. On top of that, a known kernel bug in the CFS quota code throttled containers unnecessarily and kept them from reaching even their allowed quota.
Omio disabled CPU limits until the latest kernel with the bug fix was deployed across all its clusters; disabling CFS quota in the kubelet is named as the alternative. The HTTP 5xx rate and p95 latency of a critical service dropped immediately, and the team also tracked the number of instance hours used. The cost is that a container can now take all the CPU of a node. Omio judged this safe because its automated base Dockerfiles set sane runtime defaults for its main language stacks; with poorly tuned runtimes, such as an untuned JVM from a stock image, removing limits can backfire. The write-up recommends watching utilisation, saturation and errors along with API latency and error rates after the change.
For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.
The errors appeared as readiness probe failures, container stalls, network disconnections and timeouts in calls between services. On critical services this showed as a raised rate of HTTP 5xx responses and a higher p95 response time.
The usual Kubernetes metrics did not point at the cause. Kubernetes exposes CPU usage but not the cgroup throttling counters, and unlike a memory problem, which leaves an OOMKilled restart status, CPU throttling is hard to identify. The team read it inside the pods by running cat /sys/fs/cgroup/cpu/cpu.stat and reading nr_periods (scheduling periods elapsed), nr_throttled (periods in which the container was throttled) and throttled_time (total throttled time in nanoseconds). Several applications showed throttle rates up to 50% higher than the team had assumed their limits allowed.
Root cause Software defect
Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.
A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.
IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeoutsDFDS · 2022same root cause · shared kubelet, Linux kernel
IL-0029 Slow user-facing services from CPU throttling on containers that used far less CPU than their limitBuffer · 2020same root cause · shared Container CPU limit, Linux kernel
IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six daysBarush Mendez · 2025same root cause · shared Linux kernel
IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico componentGoogle Cloud · 2022same root cause · shared kubelet
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Container CPU limit
IL-0143 GKE nodes upgraded to certain 1.13 and 1.14 versions had an elevated number of kernel panics, fixed by releases that downgraded the Container-Optimized OS imageGoogle Cloud · 2019same root cause · shared Linux kernel
IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failuresAirMap · 2018same root cause · shared kubelet
IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod evictionOath · 2018same root cause · shared kubelet