Incident Library

Organisation

Binder Project

Binder Project · binder-project

5 cases · 2017 to 2022

Root cause Software defect2Capacity exhaustion1Cloud provider outage1unknown1
Component GKE control plane2JupyterHub2Ingress controller1Kubernetes API server1Container memory limit1Node pool1Persistent volumes and stateful sets1
Contributing factor No alert on limit2Limited debug access1Harmful manual action1Monitoring blind spot1No resource requests or limits1Pages not delivered1
Symptom Service outage4Deploys failing2Container crash loop1Monitoring outage1Node NotReady1OOMKilled1Pods Pending1Pods stuck in ContainerCreating1Pods stuck in Terminating1Probe failures1Stale or inconsistent API reads1

Cases

  1. IL-0403 mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in useBinder Project · 2022

    Root cause Software defect

  2. IL-0505 kubectl logs failed on GKE with "No SSH tunnels currently open" and mybinder.org build logs stopped reaching usersBinder Project · 2019

    Root cause Cloud provider outage

  3. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the clusterBinder Project · 2018

    Root cause Software defect

  4. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVCBinder Project · 2018

    Root cause unknown

  5. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-loopedBinder Project · 2017

    Root cause Capacity exhaustion