Incident Library

IL-0402 Binder Project · 2018-03-13kubernetesgcp

mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC

Some GKE nodes of the mybinder.org cluster went bad, and the JupyterHub pod was on one of them. The pod was not deleted properly, so it never released the PVC that held the hub database, and new hub pods could not get the database, which stopped all launches. The team only noticed after several hours, because the Stackdriver alert emails went to spam and arrived hours late. After cordoning the bad nodes and deleting pods did not release the PVC, the team reset node `gke-prod-a-ssd-pool-32-134a959a-n2sk` from the GCP web interface. At 08:05 a new hub pod was scheduled on another node, pulled its image and started, and launches worked again.

Pods in "Unknown" or "NodeLost" state could not be removed with a normal delete. At 13:34 the team found that `kubectl --namespace=prod delete pod <pod-name> --grace-period=0 --force` removes them, ran it in a script for every such pod in the prod namespace, and the remaining services recovered. Why the nodes failed was left open for investigation.

Severity
–
Scope
mybinder.org could not launch any sessions for several hours; other services whose pods were on the failed nodes were down until 13:34
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On 13 March 2018 at 07:28 CET the mybinder.org team noticed that JupyterHub had not been spawning new users and that the launch success rate on the Grafana dashboard had been at zero for several hours. Many pods were in the states "Unknown" and "NodeLost", on at least the nodes `10.128.0.7` and `10.128.0.5`.

The hub pod `hub-65d9f46698-dj4jb` was in state `Unknown`, and a second hub pod, `hub-65d9f46698-dmlcv`, had been in `ContainerCreating` for 5 hours. A new hub pod could not start because the PVC holding the hub database was still claimed by the old hub pod, which was lost and did not release it. The bhub pod could not talk to the jhub pod although it could reach google.com and github.com.

Cordoning the bad nodes and deleting the two hub pods did not free the PVC, and neither did deleting every pod on node `-n2sk`. Deleting pods in "Unknown" state with a normal delete did nothing; the pods stayed listed. At 13:32 the team found that many other services were also down because their pods had been on the failed nodes and were stuck in `Unknown` or `NodeLost`.

Symptom Service outageNode NotReadyPods stuck in TerminatingPods stuck in ContainerCreating

Root cause

Root cause unknown

Component Persistent volumes and stateful setsNode pool
Contributing factor Pages not delivered
Trigger
–

Detection

The team noticed only after several hours, when the launch success rate on the Grafana dashboard had been at zero for a long time. No Stackdriver alert email arrived in time, because those emails were marked as spam and reached the team hours later.

Remediation

Action items

Lessons

The error went unnoticed for a long time because Stackdriver alert emails were marked as spam and arrived hours late. Once the problem was found, each step was logged and the steps were made more forceful as needed, so the hub was down only briefly after detection.

Sources

References

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Persistent volumes and stateful sets, Node pool among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.