Incident Library
Applied filters ComponentContainer memory limitSymptomSustained high node CPUFactorNo resource requests or limitsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
    Symptoms

    At 08:52 PM IST a report came in from WER that many students could not access the JupyterHub and that it was frozen; pages were stuck and nothing was usable. A rise in activity was visible by 09:02 PM, but the usual fixes, such as opening an incognito window and restarting servers, did not help.

    Service outageSustained high node CPU
    Trigger

    WER students' pods used a good chunk of their memory limit (tests had shown they almost always use just under 1G of their 2G limit, against a 512M guarantee) on nodes shared with the hub's core pods.

    Root cause
    Capacity exhaustion
    Component
    Node poolContainer memory limit
    Contributing factor
    No resource requests or limitsUnverified assumptionExcessive blast radius
    Detection

    WER reported stuck pages and students unable to reach the hub in a GitHub issue (2i2c-org/docs#27).

    Remediation
    • removed memory overcommit for the WER students
    • raised resource guarantees for all core pods, which restarted them and moved some to a new node
    Action items
    • Put user pods in a separate pool so they do not create pressure on the core pods (infrastructure#89)
    • Set limits on the support infrastructure (Prometheus, Grafana, ingress) as well (infrastructure#90)
    • Document and think about overcommit ratios for memory usage (infrastructure#91)
    • Set up better Grafana dashboards to monitor resource usage (infrastructure#92)
    • Document how people can get kubectl access to the cluster so others can look into issues too (infrastructure#87)
    Lessons

    Core pods without resource guarantees get whatever is left on a node, so when user pods on the same nodes fill the CPU and memory, the hub that serves everyone freezes. The memory overcommit ratio has to come from measured use: WER students used just under 1G against a 512M guarantee, which a 4x overcommit could not hold.

    View case
Check your cluster for this

This case involves Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Container memory limitNode pool
Contributing factor
Unverified assumptionExcessive blast radiusNo resource requests or limits
Organisation
2i2c
  1. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020