Cases7 cases

6 to 7 of 7

1. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest

   Zalando2018

   Symptoms

   Customers were affected, and Zalando's IAM service answered requests with HTTP 404. The talk shows this on slides titled "Customer impact" and "IAM returning 404", followed by a slide on the number of pods in the cluster.

   High error rateControl plane unresponsiveOOMKilled

   Trigger

   A CronJob (apiVersion batch/v2alpha1, schedule "*/15 9-19 * * Mon-Fri") whose concurrencyPolicy: Forbid and job history limits sat inside the pod template spec instead of the CronJob spec. The talk shows it right after the slides on the number of pods and the API server being OOM-killed, and the lessons add a quota on the number of pods.

   Root cause

   Capacity exhaustion

   Component

   Kubernetes API serverIngress controller

   Contributing factor

   Single point of failureExcessive blast radius

   Remediation

   • fixed the CronJob manifest: moved concurrencyPolicy: Forbid and the job history limits to the CronJob spec, added activeDeadlineSeconds: 120 and changed the schedule to "7 8-18 * * Mon-Fri"

   Action items

   • Make the ingress stay healthy during API server problems
   • Make the ingress keep the last known set of routes
   • Use a ResourceQuota on the number of pods (the example sets pods: "1500")

   Lessons

   An AWS ALB routes traffic to all hosts when all hosts report unhealthy, so the ingress should stay healthy during API server problems. An ingress controller that reads routes from the API server should keep its last known set of routes. Use a quota for the number of pods.

   View case
2. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down

   Universe2017

   Symptoms

   Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

   Service outageNode NotReadyNode recreation loop

   Trigger

   A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

   Root cause

   Capacity exhaustion

   Component

   Jobs and CronJobsPod scheduling and placement

   Contributing factor

   Unverified assumptionAutomation amplificationExcessive blast radius

   Detection

   The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

   Remediation

   • removed the reindex Job from the scheduler
   • let the nodes auto-heal back to healthy

   Lessons

   restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

   View case

By root cause1