Cases6 cases

5 to 6 of 6

1. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction

   Oath2018

   Symptoms

   During a rolling update of a kubelet change in one of Oath's on-premises production clusters, nodes started going NotReady, and soon all nodes in the cluster were NotReady.

   Node NotReadyPod evictionPods Pending

   Trigger

   A change to kubelet was rolled out across the nodes of one cluster; on restart, kubelet cleans up transient cgroup files, and a Kubernetes bug in that step left kubelet stuck instead of starting.

   Root cause

   Software defect

   Component

   kubeletKubernetes controller manager

   Contributing factor

   No health probesExcessive blast radius

   Detection

   Nodes reported NotReady and the eviction graphs climbed while the kubelet rollout was in progress.

   Remediation

   • failed the affected data center over so users were routed to the next available data center

   Action items

   • Run a functional test of kubelet after each upgrade batch and certify it before the next 30% of nodes are updated

   Lessons

   Check kubelet after upgrading it before moving to the next batch; a simple curl against the kubelet API would have caught this. The node controller's eviction throttling and full disruption mode, which the speakers note is not well documented, kept a cluster-wide kubelet failure from evicting every pod.

   View case
2. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down

   Universe2017

   Symptoms

   Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

   Service outageNode NotReadyNode recreation loop

   Trigger

   A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

   Root cause

   Capacity exhaustion

   Component

   Jobs and CronJobsPod scheduling and placement

   Contributing factor

   Unverified assumptionAutomation amplificationExcessive blast radius

   Detection

   The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

   Remediation

   • removed the reindex Job from the scheduler
   • let the nodes auto-heal back to healthy

   Lessons

   restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

   View case

By root cause5