Incident Library
Applied filters ComponentNode poolSymptomPod evictionOrganisation2i2cClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
    Symptoms

    On 12 May 2025 at 20:15 EEST a Freshdesk ticket reported that users' servers on the Earthscope hub were taking more than 15 minutes to start and that their kernels were being killed. It happened during a workshop and also affected other instructors using the hub; a second report said a short course of about 30 users was having major problems.

    Pod eviction
    Trigger

    A workshop with about 30 users on a profile option that guaranteed 8 GB of memory and set no memory limit on 32 GB nodes.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitNode pool
    Contributing factor
    No resource requests or limits
    Detection

    A Freshdesk support ticket from the community at 20:15 EEST, brought to the team's attention in the #support-freshdesk Slack channel at 20:49.

    Remediation
    • ran a helm rollout and hub redeploy, which partly improved server starts
    • made the 'medium' profile available and raised its memory guarantee to reduce overcommitment
    • pre-warmed the cluster with 15 nodes for the 30-user workshop
    • set explicit memory limits, with the guarantee equal to the limit, on the community's small-node profile options (https://github.com/2i2c-org/infrastructure/pull/6047)
    Action items
    • Write an incident report
    • Hold an internal retrospective about the incident (https://github.com/2i2c-org/meta/issues/2194)
    Lessons

    A memory guarantee without a limit lets a user server grow into memory that other pods on the same node were counting on, and on a shared node that ends with users' pods evicting each other. Setting the guarantee equal to the limit removes the overcommitment. For a known workshop, pre-warming enough nodes ahead of time keeps server start times short.

    View case
  2. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
    Symptoms

    The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

    Pod evictionPods Pending
    Trigger

    A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode pool
    Contributing factor
    No headroomExcessive blast radiusMonitoring blind spot
    Detection

    A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

    Action items
    • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
    • Make the Grafana OOMKiller dashboard work (#2213)
    • Move LEAP cloud infrastructure to shared nodes (#2209)
    • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
    • Decide whether Prometheus should scrape dask worker pods (#2366)
    Lessons

    Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

    View case
Check your cluster for this

These 2 cases involve Node pool and Container memory limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Node pool2Container memory limit1Prometheus1
Contributing factor
Excessive blast radius1Monitoring blind spot1No resource requests or limits1No headroom1
Organisation
2i2c2
  1. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
  2. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023