Cases16 cases

6 to 10 of 16

1. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade

   Saifeddine Rajhi2024

   Symptoms

   After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.

   OOMKilledProbe failuresMonitoring outage

   Trigger

   During the kube-prometheus-stack Helm chart upgrade, Prometheus had to replay its write-ahead log on startup, and the replay needed more memory than its limit allowed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limit

   Contributing factor

   Unknown limitNo headroom

   Detection

   The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.

   Remediation

   • ran promtool tsdb analyze inside the Prometheus pod to find high-cardinality metrics such as haproxy_server_http_responses_total

   Action items

   • Verify that the memory overhead during WAL replay stays within acceptable limits, for example 10 to 15%, and do not run Prometheus close to its memory limit
   • Optimise storage, scraping and remote write configuration and upgrade to newer Prometheus releases
   • Recover automatically from OOM crash loops, for example by deleting the WAL on OOM events
   • Enforce scrape limits so Prometheus stops scraping targets when memory use passes a threshold

   Lessons

   Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.

   View case
2. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed

   UK Ministry of Justice2024

   Symptoms

   At 12:32 Prometheus on the Cloud Platform was not available after a planned change, and the Prometheus pod had restarted several times. The platform lost its monitoring and had missing metrics for the period.

   Monitoring outageProbe failures

   Trigger

   A planned change restarted Prometheus at a time when the WAL count was high.

   Root cause

   Capacity exhaustion

   Component

   Prometheus

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   The team saw that the Prometheus pod had restarted several times after the planned change.

   Remediation

   • raised the Prometheus startupProbe threshold to 30 minutes

   Action items

   • Perform planned Prometheus restarts when the WAL count is lower
   • Set the default CPU and memory requests to meet maximum usage
   • Create a test setup that recreates the live WAL count
   • Explore the memory-snapshot-on-shutdown and auto-gomaxprocs feature flags
   • Explore storing WAL files remotely in a different location
   • Look into a blue-green Prometheus to test changes on a live-like setup before applying them to live
   • Spike into Amazon Managed Prometheus

   View case
3. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models

   OpenAI2023

   Symptoms

   Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.

   High error rateService outageProbe failures

   Trigger

   A morning with dramatically more completions than any previous day pushed the routing layer nodes, which already had a chronic memory problem, past their memory limits.

   Root cause

   Software defect

   Component

   Model request routingContainer memory limit

   Contributing factor

   No alert on limitNo headroom

   Remediation

   • limited incoming traffic
   • redeployed the routing service en masse
   • ramped traffic back up slowly

   Action items

   • Pre-allocate the responseBuffer and reuse it instead of allocating a new one in a loop (3X improvement in memory and CPU usage)
   • Adjust the configured memory limits to an appropriate level so the service has significant headroom
   • Add rate limit controls to shed traffic more gracefully
   • Increase the service's capacity
   • Change alerting to catch the underlying memory behaviour before it escalates into service issues
   • Configure autoscaling for the service, which was not possible before because added capacity could hurt an upstream service

   View case
4. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics

   UK Ministry of Justice2023

   Symptoms

   At 10:15 on 1 November 2023 a Pingdom check raised a PagerDuty high-priority alert that the "Prometheus - live healthcheck is DOWN". The same alert fired for the third time within a few minutes, and the incident was declared at 10:41.

   Monitoring outageProbe failures

   Root cause

   unknown

   Component

   Prometheus

   Contributing factor

   –

   Detection

   A Pingdom health check on the live Prometheus failed and raised a PagerDuty high-priority alert.

   Remediation

   • increased the Prometheus readiness probe timeout from 3 to 6 seconds

   Action items

   • Inspect more closely to identify this kind of failure earlier
   • Investigate whether the ingestion of data into the database is too big or too long
   • Check whether executing some queries makes Prometheus work harder and stop responding to the readiness probe
   • Check whether any other service probing Prometheus triggers the restart
   • Check whether regular Velero backups disturb EBS reads and writes and cause the restart

   View case
5. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout

   Grafana Labs2022

   Symptoms

   During a rollout of the Grafana Loki distributor Deployment in one Grafana Cloud Logs environment, the write path went down for about 25 minutes. The newly started distributor pods were OOM-killed over and over, never passed their health and readiness checks, and the rollout stopped making progress.

   Service outageOOMKilledLoad imbalanceProbe failures

   Trigger

   Step 1 of moving the Loki ring off a separately deployed key-value store onto embedded memberlist added a new label, gossip_ring_member: 'true', to the distributor pods and Service, together with a small config change, and the change required a rollout of the distributors.

   Root cause

   Configuration change

   Component

   Configuration templatingService discoveryContainer memory limit

   Contributing factor

   Unreviewed defaultsIgnored early signalUnrepresentative staging

   Detection

   The first clue was kubectl get pods and the kube_deployment_status_replicas_updated metric showing that only 16 of 140 distributor pods had been updated.

   Remediation

   • scaled the distributor Deployment out horizontally so the load spread over more pods with the new label and the rollout could continue
   • reverted the migration step to end the outage
   • set service_ignored_labels:: [$._config.gossip_member_label] and passed it to k.util.serviceFor so the label is no longer copied into the Service selector

   Action items

   • Update the function that generates a Service from a Deployment or StatefulSet to handle edge cases, including this one

   Lessons

   A Service whose selector is built by copying every label of a workload changes which pods it matches as soon as a new label is added. During a rollout it then matches only the new pods, which receive all the traffic before there are enough of them. A few errors during a rollout in other environments hid the problem, because those environments gave distributors more memory and could absorb the shift. Manifest diffs kept for every pull request made it possible to see that the Service selector had changed.

   View case

By root cause8