Cases4 cases

1 to 4 of 4

1. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use

   Binder Project2022

   Symptoms

   On 2 June 2022 from 10:00 CET, mybinder.org stopped launching any new pods, and nobody on the team noticed until a user reported a Binder outage in the Matrix channel at 21:00.

   Deploys failingService outage

   Root cause

   Software defect

   Component

   GKE control plane

   Contributing factor

   Monitoring blind spot

   Detection

   A user reported a Binder outage in the Matrix channel at 21:00, and a team member who saw it found that pods had not been launching for several hours.

   Remediation

   • Deleted the gke-resource-quotas ResourceQuota, after which pods could be created again

   Action items

   • Address uptime monitoring and alerting

   Lessons

   It took 11 hours to realise there was a major outage. Once the right object was known, deleting it fixed the problem quickly, and the team was lucky that someone with the skills and permissions to make the change was at their computer at 11pm their time.

   View case
2. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster

   Binder Project2018

   Symptoms

   On Saturday 8 July 2018 at about 18:30 CEST, the launch success rate of mybinder.org dropped to zero and stayed there. Suspicious behaviour was first reported on Gitter at 22:48, more than four hours later.

   Service outagePods PendingStale or inconsistent API reads

   Root cause

   Software defect

   Component

   JupyterHub

   Contributing factor

   No alert on limit

   Detection

   Suspicious behaviour was reported on Gitter at 22:48 CEST, more than four hours after launch success dropped to zero; no downtime notifications were set up.

   Remediation

   • Deleted user pods older than four hours with the delete-pods.py script
   • Culled pods of the most-used images, such as ipython-in-depth, more aggressively
   • Restarted the Hub pod after about 600 pods were culled, which brought launches back to normal
   • Cordoned some of the extra nodes so the autoscaler could reclaim them
   • Ran manual max-age culling the next morning, letting the cluster scale back to 3 nodes

   Action items

   • Set up notifications of downtime
   • Automate cluster scale-down
   • Document the downtime recovery steps so every team member knows them
   • Fix KubeSpawner's failure to delete pods when the pod reflector fails
   • Make KubeSpawner more robust to pod reflector failure
   • Refactor the event reflector into a single reflector instead of one per pod
   • Detect a likely unhealthy Hub state and restart the Hub without manual intervention

   Lessons

   The delete-pods.py script was useful for bulk-deleting old pods, and once the problem was noticed the cluster was healthy within a few minutes. The outage lasted five hours before anyone noticed because no downtime notifications existed, and scaling the cluster back down still took several manual steps of cordoning and draining nodes.

   View case
3. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC

   Binder Project2018

   Symptoms

   On 13 March 2018 at 07:28 CET the mybinder.org team noticed that JupyterHub had not been spawning new users and that the launch success rate on the Grafana dashboard had been at zero for several hours. Many pods were in the states "Unknown" and "NodeLost", on at least the nodes `10.128.0.7` and `10.128.0.5`.

   Service outageNode NotReadyPods stuck in TerminatingPods stuck in ContainerCreating

   Root cause

   unknown

   Component

   Persistent volumes and stateful setsNode pool

   Contributing factor

   Pages not delivered

   Detection

   The team noticed only after several hours, when the launch success rate on the Grafana dashboard had been at zero for a long time. No Stackdriver alert email arrived in time, because those emails were marked as spam and reached the team hours later.

   Remediation

   • Cordoned the bad nodes
   • Deleted the hub pod in Unknown state and the hub pod stuck in ContainerCreating
   • Reset node gke-prod-a-ssd-pool-32-134a959a-n2sk from the GCP web interface to force the PVC to be released
   • Force-deleted every pod in Unknown or NodeLost state with kubectl delete --grace-period=0 --force

   Action items

   • Investigate why the outage started, through Stackdriver logs and Google Cloud incidents
   • Set up an alerting service that does not depend on Stackdriver
   • Document how to check whether a PVC has not been released
   • Document how to restart a node by hand when commands are not working
   • Document how to delete pods that are in Unknown state

   Lessons

   The error went unnoticed for a long time because Stackdriver alert emails were marked as spam and arrived hours late. Once the problem was found, each step was logged and the steps were made more forceful as needed, so the hub was down only briefly after detection.

   View case
4. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped

   Binder Project2017

   Symptoms

   On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

   Service outageContainer crash loopProbe failuresOOMKilled

   Trigger

   A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

   Root cause

   Capacity exhaustion

   Component

   Ingress controllerContainer memory limitJupyterHub

   Contributing factor

   No resource requests or limitsHarmful manual actionNo alert on limit

   Detection

   An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

   Remediation

   • Cordoned the node that had run out of memory
   • Uncordoned the node once the proxy was working

   Action items

   • Limit the memory available to build pods
   • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
   • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
   • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod

   View case

By root cause3