Cases3 cases

1 to 3 of 3

1. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models

   OpenAI2023

   Symptoms

   Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.

   High error rateService outageProbe failures

   Trigger

   A morning with dramatically more completions than any previous day pushed the routing layer nodes, which already had a chronic memory problem, past their memory limits.

   Root cause

   Software defect

   Component

   Model request routingContainer memory limit

   Contributing factor

   No alert on limitNo headroom

   Remediation

   • limited incoming traffic
   • redeployed the routing service en masse
   • ramped traffic back up slowly

   Action items

   • Pre-allocate the responseBuffer and reuse it instead of allocating a new one in a loop (3X improvement in memory and CPU usage)
   • Adjust the configured memory limits to an appropriate level so the service has significant headroom
   • Add rate limit controls to shed traffic more gracefully
   • Increase the service's capacity
   • Change alerting to catch the underlying memory behaviour before it escalates into service issues
   • Configure autoscaling for the service, which was not possible before because added capacity could hurt an upstream service

   View case
2. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady

   UK Ministry of Justice2023

   Symptoms

   At 14:05 on 25 July 2023, a high-priority PagerDuty alert from Pingdom reported that the "Prometheus - live healthcheck" was DOWN. Prometheus on the live cluster was unavailable, and the Cloud Platform lost its monitoring for a while.

   Monitoring outageOOMKilledNode NotReady

   Trigger

   The growing number of namespaces and Prometheus rules raised Prometheus' memory needs beyond what its node instance size could hold.

   Root cause

   Capacity exhaustion

   Component

   PrometheusNode poolContainer memory limit

   Contributing factor

   No alert on limit

   Detection

   A Pingdom health check on Prometheus failed and raised a high-priority PagerDuty alert.

   Remediation

   • drained the monitoring node, moving Prometheus to another monitoring node
   • changed the node instance size to r6i.4xlarge
   • raised the Prometheus memory limit to 100Gi
   • set the Prometheus container limits to 12 CPU cores and 110Gi of memory

   Action items

   • Add an alert on node memory usage and on a pod using up most of a node's memory (issue #4538)

   View case
3. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory

   NU.nl2019

   Symptoms

   The NU.nl development team got reports that several services on one of its Kubernetes clusters behaved erratically: occasional error pages, slow responses and timeouts. The affected services were mostly internal tools and dashboards.

   Node NotReadyHigh error rateHigh latencyIntermittent timeoutsMonitoring outageOOMKilled

   Trigger

   A new ElastAlert rule, deployed shortly before from the command line, that fired if the Cloudfront-* Elasticsearch indexes stopped receiving documents; those indexes take a few million requests per hour, and the rule made ElastAlert's memory grow without bound.

   Root cause

   Configuration change

   Component

   Container memory limitkubeletPrometheus

   Contributing factor

   No resource requests or limitsUnreviewed defaultsNo alert on limitMonitoring blind spotExcessive blast radius

   Detection

   Users reported error pages, slow responses and timeouts from several services; no alert fired.

   Remediation

   • terminated the NotReady nodes one by one so the auto-scaling group replaced them (the new nodes also went NotReady, but Prometheus and Grafana landed on a stable node)
   • found the ElastAlert pods' memory growth on the Grafana pod memory dashboard
   • fixed the ElastAlert problem that caused the memory growth

   Action items

   • Define alerts on resource usage such as CPU, memory and disk space
   • Keep researching an alerting strategy that combines Elasticsearch, Prometheus Alertmanager and New Relic
   • Consider exporting metrics out of the cluster and running Grafana outside it, on EC2 or a separate Kubernetes cluster
   • Either visualise the important metrics in the ELK stack or make Prometheus and Grafana more available, and improve the metrics strategy
   • Set resource limits for ElastAlert through the Helm values
   • Configure namespaces with default memory requests and limits
   • Isolate customer-facing workloads from build, logging and monitoring workloads (a dedicated node group, or later a separate tools cluster)
   • Deploy everything through automation such as Jenkins pipelines, and consider a GitOps approach
   • Deploy through a pipeline that includes a smoke test step
   • Make Kubernetes and cloud infrastructure work part of the team's sprints, picked up by all team members while pairing with more experienced members, and hold workshops that go deep into specific topics

   Lessons

   Fixing the root cause covers only a small part of what an incident can teach; the chain of smaller failures around it is where most of the learning is. A pod without a memory limit can take whole nodes down, so limits need defaults at the namespace level. Monitoring that runs inside the cluster it watches goes blind with it. Changes installed by hand from the command line leave the rest of the team without a record of what changed.

   View case

By root cause3