IL-0403 Binder Project · 2022-06-02kubernetesgcp

mybinder.org could not create pods for hours as the GKE gke-resource-quotas quota wrongly reported 15k pods in use

A bug in the `gke-resource-quotas` ResourceQuota object that GKE manages made every new pod for the `prod` hub of mybinder.org fail with an exceeded pod quota error, although the hub had certainly not exceeded it. Launches stopped at 10:00 CET on 2 June 2022. The team learned of the outage from a user report in the Matrix channel at 21:00, found the quota error in the logs at 21:06, and at 21:22 found StackOverflow answers saying the object was buggy and that deleting it causes the object to be recreated and work correctly again. At 21:24 the team deleted `gke-resource-quotas`, pods could be created immediately, and the launch success rate went back to 100% quickly. The report also says the effect lasted about nine hours before normal operation returned without intervention.

Severity: –
Scope: The mybinder.org prod hub could not create any new pods, so no Binder sessions launched
Users affected: –
Region: –
Time to detect: 11h
Time to resolve: 11h 24m

Symptoms

On 2 June 2022 from 10:00 CET, mybinder.org stopped launching any new pods, and nobody on the team noticed until a user reported a Binder outage in the Matrix channel at 21:00.

The logs showed pod creation refused with a pod quota error: `HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"jupyter-<user>-2dpythonmoocproblems-2drkut24bk\" is forbidden: exceeded quota: gke-resource-quotas, requested: pods=1, used: pods=15k, limited: pods=15k","reason":"Forbidden","details":{"name":"jupyter-<user>-2dpythonmoocproblems-2drkut24bk","kind":"pods"},"code":403}`. The key part was `exceeded quota: gke-resource-quotas, requested: pods=1, used: pods=15k, limited: pods=15k`, which made no sense because the team was certainly not using 15k pods.

Symptom Deploys failing Service outage

Root cause

Root cause Software defect

Component GKE control plane

Contributing factor Monitoring blind spot

Trigger: –

Detection

A user reported a Binder outage in the Matrix channel at 21:00, and a team member who saw it found that pods had not been launching for several hours.

Remediation

Deleted the gke-resource-quotas ResourceQuota, after which pods could be created again

Action items

Address uptime monitoring and alerting

Lessons

It took 11 hours to realise there was a major outage. Once the right object was known, deleting it fixed the problem quickly, and the team was lucky that someone with the skills and permissions to make the change was at their computer at 11pm their time.

Sources

https://github.com/jupyterhub/mybinder.org-deploy/blob/main/docs/source/incident-reports/2022-06-02-pod-limit.md · github

References

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, GKE control plane among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0403
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0403/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.