IL-0503 Binder Project · 2017-11-30kubernetesgcp
mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
At 16:22 PST on 30 November 2017 a node in the mybinder.org cluster on GKE ran out of memory and the service started to degrade. The write-up gives a build container as the probable cause: builds had no memory limits, and one likely used all the memory on the node. That node ran the proxy pod, which holds an nginx container and a chp container; the memory exhaustion broke the proxy, its pods went into a crash loop, and launches of new servers failed with an error about creating a temporary user.
Operators noticed the problem at 16:59 because one of them happened to look at Grafana. Restarting the jupyterhub pod did not help and, because of the JupyterHub configuration, deleted all running user pods. After the OOMKilling event was found on the proxy's node, an operator suggested at 17:18 cordoning that node and then deleting the proxy pod, and the node was cordoned. At 17:20 the proxy pod was working again, although by then it had already been running for about 10 minutes, and the node was uncordoned, and at 17:21 service was restored and user pods started running again.
- Severity
- –
- Scope
- mybinder.org: new server launches failed; restarting the jupyterhub pod also removed all running user pods
- Users affected
- –
- Region
- us-central1-a
- Time to detect
- 37m
- Time to resolve
- 59m
Symptoms
On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.
The launch page showed "Found built image, launching...", "Launching server..." and then "Failed to create temporary user for gcr.io/binder-prod/r2d-<repository>:6be6d6329de99168d0a0b68f9d52228ecc61ac50". The proxy pods were in CrashloopBackoff after about 13 restarts. `kubectl describe` on them showed "Back-off restarting failed container" and "Readiness probe failed: Get http://10.12.5.36:10254/healthz: dial tcp 10.12.5.36:10254: getsockopt: connection refused" for the nginx-ingress-controller:0.9.0-beta.15 image, and the jupyterhub pod logged many DELETE requests timing out, with tracebacks.
A Google Cloud storage incident (17006) was suspected first, but all nodes showed "Ready" in the cloud console. Restarting the jupyterhub pod at 17:12 removed all user pods and the error remained. At 17:14 `kubectl describe` on the node running the proxy showed an OOMKilling event: "Memory cgroup out of memory: Kill process 8339 (python) score 1576 or sacrific".
Symptom Service outageContainer crash loopProbe failuresOOMKilled
Root cause
Root cause Capacity exhaustion
- Trigger
- A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.
Detection
An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.
Remediation
- Cordoned the node that had run out of memory
- Uncordoned the node once the proxy was working
Action items
- Limit the memory available to build pods
- Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
- Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
- Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod
Lessons
–
Sources
References
Related cases
IL-0102CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Container memory limit
IL-0548Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs laggedBuildkite · 2026same root cause · shared Container memory limit
IL-0099User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Container memory limit
IL-0170During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app servicesBackpackr · 2025same root cause · shared Ingress controller
IL-0027All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Container memory limit
IL-0044Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0016Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Container memory limit
IL-0165Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0500UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up2i2c · 2022same root cause · shared JupyterHub
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0094Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Container memory limit
IL-0093Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memoryUK Ministry of Justice · 2020same root cause · shared Ingress controller
IL-0026Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0001Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit
IL-0031Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifestZalando · 2018same root cause · shared Ingress controller
IL-0468New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchangedUC Berkeley · 2017same root cause · shared Container memory limit
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0503
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0503/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.