IL-0468 UC Berkeley · 2017-03-20kubernetesgcp
New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged
On 18 March 2017 at 16:30 the memory limit of each student server on UC Berkeley's JupyterHub on Google Container Engine was cut from 2G to 1G to use resources better. The nodes kept their size of 26G RAM, and many of them were cordoned and slowly decommissioned over the following days. As a result, the nodes reached the limit on how many disks Kubernetes would place on one node, and from some time early on 20 March new student pods stayed Pending with the scheduler reason MaxVolumeCount. Google Compute Engine itself handles more than 16 disks per node, which the team had checked, but Kubernetes did not (kubernetes/kubernetes issue 24317).
At 13:03 the team copied the instance template that Google Container Engine uses, changed the copy to spawn smaller n1-highmem-2 nodes instead of n1-highmem-4, switched the managed instance group to the new template and set it to grow by 30 nodes of about 12 students each. This left students whose servers worked undisturbed. `populate.bash` was run so student pods would start on time on the new nodes, and at 13:04 the simple autoscaler was stopped in case the mixed state of the nodes confused it. By 13:11 the new nodes were online and pods left Pending.
Because those pods started after JupyterHub's 5-minute timeout, JupyterHub lost track of them and users got 'redirected too many times'; admins stopped and started affected user pods from the control panel as users reported it. At 14:23 the hub and proxy pods were restarted, which fixed most users' state, though some requests still failed with a 599 timeout. Admins kept restarting individual users' pods by hand over the following days.
- Severity
- –
- Scope
- Some new student servers on the JupyterHub could not start and returned 500 errors from early on 20 March 2017 until about 13:00, followed by 'redirected too many times' errors, some 599 timeouts and manual restarts over the following days
- Users affected
- Students starting new servers on the JupyterHub
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
On 20 March 2017, from some time early in the day until about 13:00, some new student servers on UC Berkeley's JupyterHub stayed in Pending forever and the students got 500 errors that kept them from logging in.
At 12:44 the 500 errors were judged widespread rather than isolated. At 12:53 `kubectl describe pod` on an affected student's pod showed it Pending with "pod failed to fit in any node fit failure on node (XX): MaxVolumeCount", and all new student servers were Pending the same way. A search led to kubernetes/kubernetes issue 24317: Google Compute Engine can handle more than 16 disks per node, which the team had checked before deploying, but Kubernetes itself still could not.
After new nodes came online at 13:11 and the pods left Pending, students hit the error 'redirected too many times'. The pods had started later than the 5 minutes JupyterHub waits for a pod before giving up, so JupyterHub did not know they existed and its state no longer matched the cluster. After the hub and proxy pods were restarted at 14:23, some requests still failed with a 599 timeout, less often than in an earlier proxy incident.
Symptom Pods PendingHigh error rate
Root cause
Root cause Capacity exhaustion
- Trigger
- On 18 March at 16:30 the RAM per student was reduced from 2G to 1G to save resources while the nodes kept the same size of 26G RAM, and as a result the nodes hit the Google Cloud the limit on the number of disks per node.
Detection
New student servers reported a 500 error that kept students from logging in. At 12:44 this was judged widespread, and at 12:53 `kubectl describe pod` showed the pods Pending on MaxVolumeCount.
Remediation
- Copied the Google Container Engine instance template and changed it to spawn smaller n1-highmem-2 nodes
- Switched the managed instance group to the new template
- Grew the instance group by 30 new nodes
- Ran populate.bash so student pods would start on time on the new nodes
- Stopped the simple autoscaler
- Stopped and started affected user pods from the control panel
- Restarted the hub and proxy pods
Action items
- Follow the status of kubernetes/kubernetes issue 24317
- Find and fix the cause of the 'too many redirects' error in JupyterHub
- Delete the older, larger nodes that are no longer in use
- Alert when any pods stay Pending for a non-negligible time
Lessons
–
Sources
References
Related cases
IL-0102CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatchBuildkite · 2026same root cause · shared Node pool, Container memory limit
IL-0548Buildkite's EKS Prometheus server was killed for memory again and again, so services stopped scaling and jobs laggedBuildkite · 2026same root cause · shared Node pool, Container memory limit
IL-0099User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025same root cause · shared Node pool, Container memory limit
IL-0027All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went downBackpackr · 2025same root cause · shared Node pool, Container memory limit
IL-0016Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReadyUK Ministry of Justice · 2023same root cause · shared Node pool, Container memory limit
IL-0094Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020same root cause · shared Node pool, Container memory limit
IL-0155Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodesDigitalOcean · 2026same root cause · shared Node pool
IL-0100PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queriesPostHog · 2025same root cause · shared Node pool
IL-0082Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouseRebelMouse · 2025same root cause · shared Node pool
IL-0447An every RDS for MySQL instance slowed badly when its gp2 EBS BurstBalance ran out and IOPS fell to the baselineevery · 2024same root cause · shared Cloud block storage
IL-0044Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgradeSaifeddine Rajhi · 2024same root cause · shared Container memory limit
IL-0110A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one dayDFDS · 2024same root cause · shared Container memory limit
IL-0169The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReadyScaleway · 2024same root cause · shared Container memory limit
IL-0165Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraintsLinode · 2023same root cause · shared Container memory limit
IL-0070Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023same root cause · shared Node pool
IL-0206Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primaryGitLab · 2022same root cause · shared Container memory limit
IL-0321GCP snapshots on all Patroni nodes stalled GitLab.com's primary PostgreSQL disk, Patroni failed over and 5xx spikedGitLab · 2021same root cause · shared Cloud block storage
IL-0013The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resetsloveholidays · 2020same root cause · shared Node pool
IL-0015A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failingloveholidays · 2020same root cause · shared Node pool
IL-0026Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limitsBlue Matador · 2019same root cause · shared Container memory limit
IL-0401Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panicsKubernetes project · 2019same root cause · shared Cloud block storage
IL-0001Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queriesZalando · 2019same root cause · shared Container memory limit
IL-0518gitlab.freedesktop.org on GKE ran short of volume space and a planned 30-minute storage upgrade became a 4-hour outagefreedesktop.org · 2018same root cause · shared Cloud block storage
IL-0503mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-loopedBinder Project · 2017same root cause · shared Container memory limit
IL-0537Sakura Cloud host servers repeatedly lost their storage connection, about five minutes per port switchoverSakura Internet · 2012same root cause · shared Cloud block storage
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0468
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0468/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.