Incident Library
Applied filters ComponentPrometheusSymptomCPU throttlingFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
    Symptoms

    Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.

    CPU throttlingHigh latency
    Trigger

    Normal load on History pods that had a CPU limit of 2 cores on 8-core nodes, with the Go runtime's GOMAXPROCS left at its default of one per node core.

    Root cause
    Capacity exhaustion
    Component
    Container CPU limitLanguage runtime settingsPrometheus
    Contributing factor
    Unreviewed defaultsMonitoring blind spot
    Detection

    A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.

    Remediation
    • set the GOMAXPROCS environment variable to 2 to match the 2-core CPU limit
    Action items
    • Set GOMAXPROCS automatically from the Kubernetes CPU limit, from Temporal release 1.21.0
    Lessons

    A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.

    View case
Check your cluster for this

This case involves Container CPU limit and Language runtime settings. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Container CPU limitLanguage runtime settingsPrometheus
Contributing factor
Unreviewed defaultsMonitoring blind spot
Organisation
Temporal
  1. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023