Cases2 cases

1 to 2 of 2

1. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type

   Hyperconnect2023

   Symptoms

   Hyperconnect upgraded the Fluent Bit sidecar of one service with little user impact to Fluent Bit 2.0 and watched it for several days. Only the containers that used the Kafka output plugin were being restarted, each time on SIGSEGV. The stack trace printed at the crash pointed into the Kafka output plugin, but no core file was left behind, because Fluent Bit ran inside Kubernetes.

   Container crash loop

   Trigger

   Upgrading the Fluent Bit logging sidecar to version 2.0 (the 2.0.3 image) on a service with low user impact.

   Root cause

   Software defect

   Component

   Telemetry agent

   Contributing factor

   Limited debug access

   Detection

   The team monitored the upgraded sidecars for several days after the upgrade and saw that the containers using the Kafka output plugin were restarting with SIGSEGV.

   Remediation

   • patched the Fluent Bit Kafka output plugin to skip data that is not a map before calling produce_message()
   • ran a custom build with the patch as the sidecar, which stopped the segmentation faults
   • submitted the patch upstream as fluent-bit pull request 6452, which was merged eight days later

   Lessons

   A segmentation fault in a container can be debugged from its core file, but Kubernetes does not keep one by default: a volume mounted into the pod and a core_pattern pointing into it on the node are needed first. The debug image of the same version, run locally on the same CPU architecture with the matching source checkout, gives a gdb environment that matches production. Core files contain part of the process memory and may hold sensitive data, so they should be shared with care.

   View case
2. IL-0175 Lowering a JVM pod's CPU request from 2 to 1 pushed memory past the 90% alert because the JVM treats a cgroup v1 cpu.shares value of exactly 1024 as unset and sized Reactor Netty threads from the node's 36 CPUs

   Hyperconnect2022

   Symptoms

   During a DevOps review of resource usage across microservices, the team lowered resources.requests.cpu from 2 to 1 for notification-preferences, a Spring WebFlux service not yet used in production. A few minutes later an alarm fired because memory usage was above 90%. Neither the memory allocation nor the JVM's GC arguments had been changed.

   Memory pressure

   Trigger

   The team lowered resources.requests.cpu for the notification-preferences deployment from 2 to 1 as part of resource optimisation, with no CPU limit set.

   Root cause

   Software defect

   Component

   Language runtime settingsContainer CPU limit

   Contributing factor

   No resource requests or limits

   Detection

   A memory usage alarm above 90% fired a few minutes after the CPU request was lowered.

   Remediation

   • set -XX:ActiveProcessorCount=1 in JAVA_OPTIONS so the JVM uses one processor regardless of the cgroup values

   Action items

   • Write an internal notice about the issue

   Lessons

   On cgroup v1, a JVM in a pod with a CPU request of exactly one core and no CPU limit sees every CPU of the node, because cpu.shares 1024 is read as unset. Thread pools sized from availableProcessors(), such as Reactor Netty's I/O workers, grow accordingly and use more memory. When lowering requests for JVM services without a CPU limit, set -XX:ActiveProcessorCount or a CPU limit explicitly. Later JDKs ignore CPU requests for this calculation altogether.

   View case

By root cause1