Cases32 cases

31 to 32 of 32

1. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state

   Nordstrom2017

   Symptoms

   A user reported that pods were starting up and then vanishing. When the cluster operators looked, kubectl get pods and kubectl get nodes returned two different sets of data that alternated back and forth, as if there were two opinions of the cluster's state.

   Stale or inconsistent API readsStale endpoint routingService outage

   Trigger

   etcd write latency rose far above the healthy single-digit milliseconds; the speakers say trouble starts once the database sync time crosses double digits or several hundred milliseconds.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverIngress controllerPersistent volumes and stateful sets

   Contributing factor

   Unreviewed defaults

   Detection

   A user reported pods starting up and vanishing.

   Remediation

   • restored service on a replacement cluster
   • freed persistent volumes from the old cluster one by one so they could bind in the new cluster
   • updated DNS records that pointed at the old cluster's load balancer names

   Action items

   • Make the Kubernetes API server request quorum reads from etcd

   Lessons

   etcd is extremely sensitive to write latency; once the database sync time crosses double digits of milliseconds, stale reads or badly degraded performance follow. Without quorum reads, a consistent store can still hand out stale data. Documentation for a fast-moving system changes, so read it again regularly. Moving workloads to a new cluster is hard when they hold persistent volumes or when DNS records point at generated load balancer names.

   View case
2. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped

   Binder Project2017

   Symptoms

   On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

   Service outageContainer crash loopProbe failuresOOMKilled

   Trigger

   A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

   Root cause

   Capacity exhaustion

   Component

   Ingress controllerContainer memory limitJupyterHub

   Contributing factor

   No resource requests or limitsHarmful manual actionNo alert on limit

   Detection

   An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

   Remediation

   • Cordoned the node that had run out of memory
   • Uncordoned the node once the proxy was working

   Action items

   • Limit the memory available to build pods
   • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
   • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
   • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod

   View case

By root cause8