Cases11 cases

10 to 11 of 11

1. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down

   Grafana Labs2019

   Symptoms

   On Friday 19 July 2019 the Grafana Cloud Hosted Prometheus service, which is built on the Cortex project, was down for 26 minutes. New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start.

   Pod evictionHigh error rateService outageOOMKilled

   Trigger

   An engineer created a new dedicated Cortex cluster for a large customer. Its configuration did not include the new pod priorities, so all its pods got the default medium priority, and the Kubernetes cluster did not have enough free resources to fit it.

   Root cause

   Configuration change

   Component

   Pod priority and preemption

   Contributing factor

   Partial rolloutAutomation amplificationMissed in reviewNo headroomRetry storm

   Detection

   New error-budget-based Prometheus alerts paged the on-call engineer within 4 minutes of the start of the outage.

   Remediation

   • Scaled up the underlying Kubernetes cluster to fit both the new and the production Cortex clusters
   • Diagnosed and fixed the out-of-memory errors in the authenticating reverse proxies in front of Cortex

   Action items

   • Give the production ingesters the high priority class, which should have been done before medium became the default
   • Have more people review changes to configuration that is global to the cluster
   • Require a design document for every configuration change outside a namespace
   • Automate the sizing of the frontend authenticating reverse proxy so it does not run out of memory under overload
   • Investigate the default backoff and scaling behaviour of Prometheus to prevent a stampeding herd of retries

   Lessons

   Medium should not have become the default priority until the production ingesters had the high priority, and that designation should have reached production sooner. The change was considered too minor for a design document and was only discussed in a GitHub issue; cluster-global configuration needs more scrutiny. On the positive side, once Cortex had enough capacity it recovered by itself, and Grafana Loki helped confirm that the ingesters exited gracefully.

   View case
2. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelApplication database

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case

By root cause6