Cases6 cases

1 to 5 of 6

1. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade

   Saifeddine Rajhi2024

   Symptoms

   After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.

   OOMKilledProbe failuresMonitoring outage

   Trigger

   During the kube-prometheus-stack Helm chart upgrade, Prometheus had to replay its write-ahead log on startup, and the replay needed more memory than its limit allowed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limit

   Contributing factor

   Unknown limitNo headroom

   Detection

   The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.

   Remediation

   • ran promtool tsdb analyze inside the Prometheus pod to find high-cardinality metrics such as haproxy_server_http_responses_total

   Action items

   • Verify that the memory overhead during WAL replay stays within acceptable limits, for example 10 to 15%, and do not run Prometheus close to its memory limit
   • Optimise storage, scraping and remote write configuration and upgrade to newer Prometheus releases
   • Recover automatically from OOM crash loops, for example by deleting the WAL on OOM events
   • Enforce scrape limits so Prometheus stops scraping targets when memory use passes a threshold

   Lessons

   Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.

   View case
2. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed

   UK Ministry of Justice2024

   Symptoms

   At 12:32 Prometheus on the Cloud Platform was not available after a planned change, and the Prometheus pod had restarted several times. The platform lost its monitoring and had missing metrics for the period.

   Monitoring outageProbe failures

   Trigger

   A planned change restarted Prometheus at a time when the WAL count was high.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   The team saw that the Prometheus pod had restarted several times after the planned change.

   Remediation

   • raised the Prometheus startupProbe threshold to 30 minutes

   Action items

   • Perform planned Prometheus restarts when the WAL count is lower
   • Set the default CPU and memory requests to meet maximum usage
   • Create a test setup that recreates the live WAL count
   • Explore the memory-snapshot-on-shutdown and auto-gomaxprocs feature flags
   • Explore storing WAL files remotely in a different location
   • Look into a blue-green Prometheus to test changes on a live-like setup before applying them to live
   • Spike into Amazon Managed Prometheus

   View case
3. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady

   UK Ministry of Justice2023

   Symptoms

   At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

   Monitoring outageOOMKilledNode NotReady

   Trigger

   The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode poolContainer memory limit

   Contributing factor

   No alert on limit

   Detection

   A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

   Remediation

   • drained the monitoring node, moving Prometheus to another monitoring node
   • changed the node instance size to r6i.4xlarge
   • raised the Prometheus memory limit to 100Gi
   • set the Prometheus container limits to 12 CPU cores and 110Gi of memory

   Action items

   • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)

   View case
4. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent

   Temporal2023

   Symptoms

   Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.

   CPU throttlingHigh latency

   Trigger

   Normal load on History pods that had a CPU limit of 2 cores on 8-core nodes, with the Go runtime's GOMAXPROCS left at its default of one per node core.

   Root cause

   Capacity exhaustion

   Component

   Container CPU limitLanguage runtime settingsPrometheus

   Contributing factor

   Unreviewed defaultsMonitoring blind spot

   Detection

   A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.

   Remediation

   • set the GOMAXPROCS environment variable to 2 to match the 2-core CPU limit

   Action items

   • Set GOMAXPROCS automatically from the Kubernetes CPU limit, from Temporal release 1.21.0

   Lessons

   A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.

   View case
5. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up

   2i2c2023

   Symptoms

   The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

   Pod evictionPods Pending

   Trigger

   A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode pool

   Contributing factor

   No headroomExcessive blast radiusMonitoring blind spot

   Detection

   A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

   Action items

   • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
   • Make the Grafana OOMKiller dashboard work (#2213)
   • Move LEAP cloud infrastructure to shared nodes (#2209)
   • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
   • Decide whether Prometheus should scrape dask worker pods (#2366)

   Lessons

   Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

   View case

By root cause1