Cases3 cases

1 to 3 of 3

1. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub

   2i2c2020

   Symptoms

   At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

   Service outageSustained high node CPU

   Trigger

   WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

   Root cause

   Capacity exhaustion

   Component

   Node poolContainer memory limit

   Contributing factor

   No resource requests or limitsUnverified assumptionExcessive blast radius

   Detection

   WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

   Remediation

   • removed memory overcommit for the WER students
   • raised resource guarantees for all core pods, which restarted them and moved some to a new node

   Action items

   • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
   • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
   • Document and think about overcommit ratios for memory usage (infrastructure#91)
   • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
   • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)

   Lessons

   Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

   View case
2. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down

   Moonlight2019

   Symptoms

   On Friday, January 18, Moonlight's API began returning occasional errors that looked like connection problems with its Redis database. The API checks Redis on every authenticated request to validate the session. At the same time the Kubernetes monitoring reported some nodes and pods as unresponsive.

   High error rateIntermittent connection failuresNode NotReadyService outageSustained high node CPU

   Trigger

   The Kubernetes scheduler put several CPU-heavy pods on the same node, including all three replicas of the website and the server-side rendering pods.

   Root cause

   Capacity exhaustion

   Component

   Pod scheduling and placementLinux kernelApplication database

   Contributing factor

   Single point of failureUnverified assumptionAutomation amplification

   Detection

   API errors that looked like Redis connectivity problems, and Kubernetes monitoring that showed nodes and pods as unresponsive. Google Cloud Support later found the link between sustained 100% CPU and kernel panics.

   Remediation

   • added anti-affinity rules to all major deployments so their pods spread across nodes

   Action items

   • Keep working with Google Cloud support to find and fix the cause of the node kernel panics

   Lessons

   Three replicas give no fault tolerance if the scheduler places them all on one node; anti-affinity rules make the spread explicit. A node failure that comes from the workload itself moves with the workload when the scheduler reschedules it.

   View case
3. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down

   Universe2017

   Symptoms

   Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

   Service outageNode NotReadyNode recreation loop

   Trigger

   A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

   Root cause

   Capacity exhaustion

   Component

   Jobs and CronJobsPod scheduling and placement

   Contributing factor

   Unverified assumptionAutomation amplificationExcessive blast radius

   Detection

   The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

   Remediation

   • removed the reindex Job from the scheduler
   • let the nodes auto-heal back to healthy

   Lessons

   restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

   View case

By root cause1