Cases2 cases

1 to 2 of 2

1. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent

   Temporal2023

   Symptoms

   Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.

   CPU throttlingHigh latency

   Trigger

   Normal load on History pods that had a CPU limit of 2 cores on 8-core nodes, with the Go runtime's GOMAXPROCS left at its default of one per node core.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitLanguage runtime settingsPrometheus

   Contributing factor

   Unreviewed defaultsMonitoring blind spot

   Detection

   A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.

   Remediation

   • set the GOMAXPROCS environment variable to 2 to match the 2-core CPU limit

   Action items

   • Set GOMAXPROCS automatically from the Kubernetes CPU limit, from Temporal release 1.21.0

   Lessons

   A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.

   View case
2. IL-0175 Lowering a JVM pod's CPU request from 2 to 1 pushed memory past the 90% alert because the JVM treats a cgroup v1 cpu.shares value of exactly 1024 as unset and sized Reactor Netty threads from the node's 36 CPUs

   Hyperconnect2022

   Symptoms

   During a DevOps review of resource usage across microservices, the team lowered resources.requests.cpu from 2 to 1 for notification-preferences, a Spring WebFlux service not yet used in production. A few minutes later an alarm fired because memory usage was above 90%. Neither the memory allocation nor the JVM's GC arguments had been changed.

   Memory pressure

   Trigger

   The team lowered resources.requests.cpu for the notification-preferences deployment from 2 to 1 as part of resource optimisation, with no CPU limit set.

   Root cause

   Software defect

   Component

   Language runtime settingsContainer CPU limit

   Contributing factor

   No resource requests or limits

   Detection

   A memory usage alarm above 90% fired a few minutes after the CPU request was lowered.

   Remediation

   • set -XX:ActiveProcessorCount=1 in JAVA_OPTIONS so the JVM uses one processor regardless of the cgroup values

   Action items

   • Write an internal notice about the issue

   Lessons

   On cgroup v1, a JVM in a pod with a CPU request of exactly one core and no CPU limit sees every CPU of the node, because cpu.shares 1024 is read as unset. Thread pools sized from availableProcessors(), such as Reactor Netty's I/O workers, grow accordingly and use more memory. When lowering requests for JVM services without a CPU limit, set -XX:ActiveProcessorCount or a CPU limit explicitly. Later JDKs ignore CPU requests for this calculation altogether.

   View case

By root cause2