Cases1 case
1 to 1 of 1
IL-0402mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVCBinder Project2018- Symptoms
On 13 March 2018 at 07:28 CET the mybinder.org team noticed that JupyterHub had not been spawning new users and that the launch success rate on the Grafana dashboard had been at zero for several hours. Many pods were in the states "Unknown" and "NodeLost", on at least the nodes `10.128.0.7` and `10.128.0.5`.
- Root cause
- Component
- Contributing factor
- Detection
The team noticed only after several hours, when the launch success rate on the Grafana dashboard had been at zero for a long time. No Stackdriver alert email arrived in time, because those emails were marked as spam and reached the team hours later.
- Remediation
- Cordoned the bad nodes
- Deleted the hub pod in Unknown state and the hub pod stuck in ContainerCreating
- Reset node gke-prod-a-ssd-pool-32-134a959a-n2sk from the GCP web interface to force the PVC to be released
- Force-deleted every pod in Unknown or NodeLost state with kubectl delete --grace-period=0 --force
- Action items
- Investigate why the outage started, through Stackdriver logs and Google Cloud incidents
- Set up an alerting service that does not depend on Stackdriver
- Document how to check whether a PVC has not been released
- Document how to restart a node by hand when commands are not working
- Document how to delete pods that are in Unknown state
- Lessons
The error went unnoticed for a long time because Stackdriver alert emails were marked as spam and arrived hours late. Once the problem was found, each step was logged and the steps were made more forceful as needed, so the hub was down only briefly after detection.
This case involves Node pool and Persistent volumes and stateful sets. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.