Incident Library
Applied filters ComponentNode poolSymptomPod evictionFactorNo resource requests or limitsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
    Symptoms

    On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.

    Pod eviction
    Trigger

    A workshop with about 30 users on a profile option that guaranteed 8 GB of memory and set no memory limit on 32 GB nodes.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitNode pool
    Contributing factor
    No resource requests or limits
    Detection

    A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.

    Remediation
    • ran a helm rollout and hub redeploy, which partly improved server starts
    • made the 'medium' profile available and raised its memory guarantee to reduce overcommitment
    • pre-warmed the cluster with 15 nodes for the 30-user workshop
    • set explicit memory limits, with the guarantee equal to the limit, on the community's small-node profile options (https://github.com/2i2c-org/infrastructure/pull/6047)
    Action items
    • Write an incident report
    • Hold an internal retrospective about the incident (https://github.com/2i2c-org/meta/issues/2194)
    Lessons

    A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment. For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.

    View case
Check your cluster for this

This case involves Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Container memory limitNode pool
Contributing factor
No resource requests or limits
Organisation
2i2c
  1. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025