Cases2 cases

1 to 2 of 2

1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day

   DFDS2024

   Symptoms

   At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

   Node NotReadyContainer crash loopOOMKilledMonitoring outage

   Trigger

   Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

   Root cause

   Capacity exhaustion

   Component

   Telemetry agentContainer memory limitApplication database

   Contributing factor

   Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot

   Detection

   A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

   Remediation

   • marked the EC2 instance unhealthy so a new node started
   • cordoned and force-drained the NotReady node and terminated its EC2 instance
   • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
   • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
   • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
   • emailed the team that owns the dev container, which confirmed it was working on a fix

   Action items

   • Move grafana-agent to the monitoring node group
   • Add alerts for when bandwidth across the cluster reaches a certain threshold
   • Investigate whether memory limits can be set on the grafana-agent Helm chart
   • Investigate grafana-agent memory usage
   • Ask Grafana support whether grafana-agent should use a PVC
   • Consider pushing node logs to Loki with limited retention
   • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes

   Lessons

   Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

   View case
2. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type

   Hyperconnect2023

   Symptoms

   Hyperconnect upgraded the Fluent Bit sidecar of one service with little user impact to Fluent Bit 2.0 and watched it for several days. Only the containers that used the Kafka output plugin were being restarted, each time on SIGSEGV. The stack trace printed at the crash pointed into the Kafka output plugin, but no core file was left behind, because Fluent Bit ran inside Kubernetes.

   Container crash loop

   Trigger

   Upgrading the Fluent Bit logging sidecar to version 2.0 (the 2.0.3 image) on a service with low user impact.

   Root cause

   Software defect

   Component

   Telemetry agent

   Contributing factor

   Limited debug access

   Detection

   The team monitored the upgraded sidecars for several days after the upgrade and saw that the containers using the Kafka output plugin were restarting with SIGSEGV.

   Remediation

   • patched the Fluent Bit Kafka output plugin to skip data that is not a map before calling produce_message()
   • ran a custom build with the patch as the sidecar, which stopped the segmentation faults
   • submitted the patch upstream as fluent-bit pull request 6452, which was merged eight days later

   Lessons

   A segmentation fault in a container can be debugged from its core file, but Kubernetes does not keep one by default: a volume mounted into the pod and a core_pattern pointing into it on the node are needed first. The debug image of the same version, run locally on the same CPU architecture with the matching source checkout, gives a gdb environment that matches production. Core files contain part of the process memory and may hold sensitive data, so they should be shared with care.

   View case

By root cause2