Incident Library
Applied filters Root causeunknownComponentPersistent volumes and stateful setsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC
    Binder Project2018
    Symptoms

    On 13 March 2018 at 07:28 CET the mybinder.org team noticed that JupyterHub had not been spawning new users and that the launch success rate on the Grafana dashboard had been at zero for several hours. Many pods were in the states "Unknown" and "NodeLost", on at least the nodes `10.128.0.7` and `10.128.0.5`.

    Service outageNode NotReadyPods stuck in TerminatingPods stuck in ContainerCreating
    Root cause
    unknown
    Component
    Persistent volumes and stateful setsNode pool
    Contributing factor
    Pages not delivered
    Detection

    The team noticed only after several hours, when the launch success rate on the Grafana dashboard had been at zero for a long time. No Stackdriver alert email arrived in time, because those emails were marked as spam and reached the team hours later.

    Remediation
    • Cordoned the bad nodes
    • Deleted the hub pod in Unknown state and the hub pod stuck in ContainerCreating
    • Reset node gke-prod-a-ssd-pool-32-134a959a-n2sk from the GCP web interface to force the PVC to be released
    • Force-deleted every pod in Unknown or NodeLost state with kubectl delete --grace-period=0 --force
    Action items
    • Investigate why the outage started, through Stackdriver logs and Google Cloud incidents
    • Set up an alerting service that does not depend on Stackdriver
    • Document how to check whether a PVC has not been released
    • Document how to restart a node by hand when commands are not working
    • Document how to delete pods that are in Unknown state
    Lessons

    The error went unnoticed for a long time because Stackdriver alert emails were marked as spam and arrived hours late. Once the problem was found, each step was logged and the steps were made more forceful as needed, so the hub was down only briefly after detection.

    View case
Check your cluster for this

This case involves Node pool and Persistent volumes and stateful sets. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

unknown1 case

Component
Node poolPersistent volumes and stateful sets
Contributing factor
Pages not delivered
Organisation
Binder Project
  1. IL-0402 mybinder.org launches dropped to zero for hours because a JupyterHub pod on a lost GKE node kept its hub database PVC
    Binder Project2018