Cases13 cases

6 to 10 of 13

1. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade

   Saifeddine Rajhi2024

   Symptoms

   After an upgrade of the kube-prometheus-stack Helm chart, several pods stayed not ready. Among them was prometheus-prometheus-operator-prometheus-0, showing 3/4 Running and a recent termination.

   OOMKilledProbe failuresMonitoring outage

   Trigger

   During the kube-prometheus-stack Helm chart upgrade, Prometheus had to replay its write-ahead log on startup, and the replay needed more memory than its limit allowed.

   Root cause

   Capacity exhaustion

   Component

   PrometheusContainer memory limit

   Contributing factor

   Unknown limitNo headroom

   Detection

   The team noticed pods not ready after the Helm chart upgrade and found the OOMKilled termination with kubectl describe pod.

   Remediation

   • ran promtool tsdb analyze inside the Prometheus pod to find high-cardinality metrics such as haproxy_server_http_responses_total

   Action items

   • Verify that the memory overhead during WAL replay stays within acceptable limits, for example 10 to 15%, and do not run Prometheus close to its memory limit
   • Optimise storage, scraping and remote write configuration and upgrade to newer Prometheus releases
   • Recover automatically from OOM crash loops, for example by deleting the WAL on OOM events
   • Enforce scrape limits so Prometheus stops scraping targets when memory use passes a threshold

   Lessons

   Size Prometheus for its restart, not only for steady state: WAL replay can need far more memory than normal running, and a Prometheus that already runs near its limit can be OOM-killed during the replay. Much of Prometheus memory use comes from cardinality, which promtool tsdb analyze can measure even when Prometheus itself is down.

   View case
2. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load

   Google Cloud2024

   Symptoms

   From 12:06 US/Pacific on 12 June 2024, services built on Vertex AI Prediction began reporting failed user requests in eight regions: us-central1, asia-southeast1, europe-west3, europe-west4, us-east1, us-west1, northamerica-northeast1 and us-east4.

   High error rateHigh latencyIntermittent timeouts

   Trigger

   Starting on 8 June 2024, a new form of user request to the GenerateContent API caused intermittent segmentation faults in the Vertex Serving Prediction API servers; on 12 June the rate of such requests increased.

   Root cause

   Software defect

   Component

   Model inference serverCloud load balancer

   Contributing factor

   Partial rolloutNo headroom

   Detection

   Internal production monitoring alerted engineers to the segfaults on 10 June 2024 at 07:35 US/Pacific, before any visible customer impact; on 12 June services built on Vertex Prediction reported failing user requests.

   Remediation

   • confirmed the new failures came from the same segfault found on 10 June
   • accelerated the rollout of the fix that was already in progress, completed at 15:59 US/Pacific

   Action items

   • Ensure early signals of server binary issues are captured in production health analysis and investigated in time
   • Ensure release validation of feature changes and updates in production

   Lessons

   A crash that only some requests trigger can stay invisible while load balancers route around the restarting servers. It becomes an outage when the share of such requests grows until too many servers are restarting at once. Here the cause was known and a fix was rolling out two days before customers were affected, but the progressive rollout had not finished when the triggering traffic increased; Google's follow-up is to investigate early signals of server binary issues in time.

   View case
3. IL-0024 Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three times

   Hugging Face2024

   Symptoms

   At 08:45 CET on 22 April 2024 the Hugging Face Hub became unstable, and by 09:30 it was unresponsive. For most users the website was slow or could not be reached at all.

   Service outageHigh latencyDatabase overload

   Trigger

   Bursts of requests to revision endpoints that read heavily from the *_info_cache collections: a call to https://huggingface.co/api/datasets/lighteval/mmlu/revision/main rose sharply right before the first crash, and a call to https://huggingface.co/api/models/sentence-transformers/all-mpnet-base-v2/revision/main, a model with 300+ associated spaces, lined up with the crash at 21:30 on 23 April.

   Root cause

   Software defect

   Component

   Application databaseModel hubCluster autoscaler

   Contributing factor

   No headroomRetry stormOverlapping failures

   Detection

   Instability started at 08:45 on 22 April and the Hub was unresponsive by 09:30; the team followed MongoDB Atlas metrics and a MongoDB support case. The root cause was found days later from a new graph of requests per URL.

   Remediation

   • Added a firewall rule blocking all requests with a Java/* user agent
   • Disabled the quicksearch feature on the Hub
   • Scaled the MongoDB Atlas cluster by hand from tier 2 to tier 3
   • Switched the Hub to maintenance mode with a 503 page through AWS WAF and blocked incoming traffic
   • Scaled all Hub pods down to 0 and brought them back
   • Set an aggressive rate limit
   • Added timeouts to requests and reworked database queries that were slow or returned large bodies

   Action items

   • Add timeouts to requests based on their usual completion time
   • Optimise requests that took too long or returned a large body, and rework poorly designed database queries
   • Remove old unused indexes from the database
   • Build new dashboards and monitoring metrics
   • Move the *_info_cache collections to Redis
   • Make it possible to abort any database request when the client aborts theirs

   Lessons

   Finding the root cause took two to three days: the existing charts did not show it, and a per-URL request breakdown did. Not cancelling database work when a client goes away turns a strained database into a thundering herd. A close inspection of the database exposed architectural flaws and bottlenecks, such as a collection with large unused indexes, and taught the team which metrics need closer watching to anticipate crashes. Unrelated failures happening at the same time added to the stress and made restoring service harder.

   View case
4. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models

   OpenAI2023

   Symptoms

   Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.

   High error rateService outageProbe failures

   Trigger

   A morning with dramatically more completions than any previous day pushed the routing layer nodes, which already had a chronic memory problem, past their memory limits.

   Root cause

   Software defect

   Component

   Model request routingContainer memory limit

   Contributing factor

   No alert on limitNo headroom

   Remediation

   • limited incoming traffic
   • redeployed the routing service en masse
   • ramped traffic back up slowly

   Action items

   • Pre-allocate the responseBuffer and reuse it instead of allocating a new one in a loop (3X improvement in memory and CPU usage)
   • Adjust the configured memory limits to an appropriate level so the service has significant headroom
   • Add rate limit controls to shed traffic more gracefully
   • Increase the service's capacity
   • Change alerting to catch the underlying memory behaviour before it escalates into service issues
   • Configure autoscaling for the service, which was not possible before because added capacity could hurt an upstream service

   View case
5. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up

   2i2c2023

   Symptoms

   The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

   Pod evictionPods Pending

   Trigger

   A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode pool

   Contributing factor

   No headroomExcessive blast radiusMonitoring blind spot

   Detection

   A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

   Action items

   • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
   • Make the Grafana OOMKiller dashboard work (#2213)
   • Move LEAP cloud infrastructure to shared nodes (#2209)
   • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
   • Decide whether Prometheus should scrape dask worker pods (#2366)

   Lessons

   Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

   View case

By root cause8