Incident Library
Applied filters ComponentLinux kernelSymptomCPU throttlingFactorUnknown limitClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
    Symptoms

    For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

    CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures
    Trigger

    Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

    Root cause
    Software defect
    Component
    Container CPU limitLinux kernelkubelet
    Contributing factor
    Unknown limitMonitoring blind spotUnpatched known bug
    Detection

    Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

    Remediation
    • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters
    Action items
    • Deploy the kernel with the CFS quota bug fix across all clusters
    • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected
    Lessons

    A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

    View case
Check your cluster for this

This case involves Container CPU limit and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Container CPU limitkubeletLinux kernel
Contributing factor
Unpatched known bugUnknown limitMonitoring blind spot
Organisation
Omio
  1. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020