Incident Library

IL-0504 Binder Project · 2018-07-08kubernetes

mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster

At about 18:30 CEST the pod reflector in KubeSpawner stopped receiving events, and every launch failed until the Hub was restarted. A second bug made it worse: pods that had started but were not registered in the reflector had their launch aborted, and KubeSpawner then skipped deleting them because it assumed those pods had never been created. The write-up names this skipped deletion as the direct cause of the runaway pod growth and the reflector that stopped updating as the root cause. Why the reflector stopped is not established; the new event reflectors, a major new KubeSpawner feature, may have been relevant.

There were no downtime notifications, so the outage ran for hours before anyone noticed. From 23:25 the team deleted pods older than four hours with the delete-pods.py script, culled pods of the most-used images such as ipython-in-depth more aggressively, and restarted the Hub pod; after about 600 pods were culled, launches were back to normal within a few minutes. Some of the nodes added for the flood of pods were cordoned so the autoscaler could reclaim them, and the next morning manual max-age culling let the cluster scale back down to 3 nodes. The total service disruption lasted about five hours.

Severity
–
Scope
mybinder.org: all new launches failed; the cluster filled with almost 1k user pods
Users affected
all mybinder.org users launching servers
Region
–
Time to detect
4h 18m
Time to resolve
5h

Symptoms

On Saturday 8 July 2018 at about 18:30 CEST, the launch success rate of mybinder.org dropped to zero and stayed there. Suspicious behaviour was first reported on Gitter at 22:48, more than four hours later.

When the investigation began at 23:25, the cluster was full and had scaled up to 8 nodes. It held almost 1k user pods: 737 running and 240 waiting to be scheduled.

The hub logs, read the next morning, showed that the pod reflector had stopped receiving events at about 18:30, with errors such as "TimeoutError: pod/jupyter-ipython-2dipython-2din-2ddepth-... did not disappear in 300 seconds!" and "TimeoutError: pod/jupyter-bokeh-2dbokeh-2dnotebooks-... did not start in 300 seconds!". Shortly after came warnings that deletion was skipped for pods that had launched: "No pod jupyter-ipython-2dipython-2din-2ddepth-... to delete. Assuming already deleted."

Symptom Service outagePods PendingStale or inconsistent API reads

Root cause

Root cause Software defect

Component JupyterHub
Contributing factor No alert on limit
Trigger
–

Detection

Suspicious behaviour was reported on Gitter at 22:48 CEST, more than four hours after launch success dropped to zero; no downtime notifications were set up.

Remediation

Action items

Lessons

The delete-pods.py script was useful for bulk-deleting old pods, and once the problem was noticed the cluster was healthy within a few minutes. The outage lasted five hours before anyone noticed because no downtime notifications existed, and scaling the cluster back down still took several manual steps of cordoning and draining nodes.

Sources

References

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, JupyterHub among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.