Cases1 case

1 to 1 of 1

1. IL-0175 Lowering a JVM pod's CPU request from 2 to 1 pushed memory past the 90% alert because the JVM treats a cgroup v1 cpu.shares value of exactly 1024 as unset and sized Reactor Netty threads from the node's 36 CPUs

   Hyperconnect2022

   Symptoms

   During a DevOps review of resource usage across microservices, the team lowered resources.requests.cpu from 2 to 1 for notification-preferences, a Spring WebFlux service not yet used in production. A few minutes later an alarm fired because memory usage was above 90%. Neither the memory allocation nor the JVM's GC arguments had been changed.

   Memory pressure

   Trigger

   The team lowered resources.requests.cpu for the notification-preferences deployment from 2 to 1 as part of resource optimisation, with no CPU limit set.

   Root cause

   Software defect

   Component

   Language runtime settingsContainer CPU limit

   Contributing factor

   No resource requests or limits

   Detection

   A memory usage alarm above 90% fired a few minutes after the CPU request was lowered.

   Remediation

   • set -XX:ActiveProcessorCount=1 in JAVA_OPTIONS so the JVM uses one processor regardless of the cgroup values

   Action items

   • Write an internal notice about the issue

   Lessons

   On cgroup v1, a JVM in a pod with a CPU request of exactly one core and no CPU limit sees every CPU of the node, because cpu.shares 1024 is read as unset. Thread pools sized from availableProcessors(), such as Reactor Netty's I/O workers, grow accordingly and use more memory. When lowering requests for JVM services without a CPU limit, set -XX:ActiveProcessorCount or a CPU limit explicitly. Later JDKs ignore CPU requests for this calculation altogether.

   View case

By root cause1