Cases2 cases

1 to 2 of 2

1. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster

   Binder Project2018

   Symptoms

   On Saturday 8 July 2018 at about 18:30 CEST, the launch success rate of mybinder.org dropped to zero and stayed there. Suspicious behaviour was first reported on Gitter at 22:48, more than four hours later.

   Service outagePods PendingStale or inconsistent API reads

   Root cause

   Software defect

   Component

   JupyterHub

   Contributing factor

   No alert on limit

   Detection

   Suspicious behaviour was reported on Gitter at 22:48 CEST, more than four hours after launch success dropped to zero; no downtime notifications were set up.

   Remediation

   • Deleted user pods older than four hours with the delete-pods.py script
   • Culled pods of the most-used images, such as ipython-in-depth, more aggressively
   • Restarted the Hub pod after about 600 pods were culled, which brought launches back to normal
   • Cordoned some of the extra nodes so the autoscaler could reclaim them
   • Ran manual max-age culling the next morning, letting the cluster scale back to 3 nodes

   Action items

   • Set up notifications of downtime
   • Automate cluster scale-down
   • Document the downtime recovery steps so every team member knows them
   • Fix KubeSpawner's failure to delete pods when the pod reflector fails
   • Make KubeSpawner more robust to pod reflector failure
   • Refactor the event reflector into a single reflector instead of one per pod
   • Detect a likely unhealthy Hub state and restart the Hub without manual intervention

   Lessons

   The delete-pods.py script was useful for bulk-deleting old pods, and once the problem was noticed the cluster was healthy within a few minutes. The outage lasted five hours before anyone noticed because no downtime notifications existed, and scaling the cluster back down still took several manual steps of cordoning and draining nodes.

   View case
2. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state

   Nordstrom2017

   Symptoms

   A user reported that pods were starting up and then vanishing. When the cluster operators looked, kubectl get pods and kubectl get nodes returned two different sets of data that alternated back and forth, as if there were two opinions of the cluster's state.

   Stale or inconsistent API readsStale endpoint routingService outage

   Trigger

   etcd write latency rose far above the healthy single-digit milliseconds; the speakers say trouble starts once the database sync time crosses double digits or several hundred milliseconds.

   Root cause

   Software defect

   Component

   etcdKubernetes API serverIngress controllerPersistent volumes and stateful sets

   Contributing factor

   Unreviewed defaults

   Detection

   A user reported pods starting up and vanishing.

   Remediation

   • restored service on a replacement cluster
   • freed persistent volumes from the old cluster one by one so they could bind in the new cluster
   • updated DNS records that pointed at the old cluster's load balancer names

   Action items

   • Make the Kubernetes API server request quorum reads from etcd

   Lessons

   etcd is extremely sensitive to write latency; once the database sync time crosses double digits of milliseconds, stale reads or badly degraded performance follow. Without quorum reads, a consistent store can still hand out stale data. Documentation for a fast-moving system changes, so read it again regularly. Moving workloads to a new cluster is hard when they hold persistent volumes or when DNS records point at generated load balancer names.

   View case

By root cause1