Incident Library

Organisation

2i2c

2i2c · 2i2c

7 cases · 2020 to 2026

Root cause Capacity exhaustion3Breaking change in an upgrade2Cloud provider outage1Operator error1
Component Node pool4Cluster autoscaler2Ingress controller2Container memory limit2cert-manager1Jobs and CronJobs1Persistent volumes and stateful sets1Prometheus1Terraform1
Contributing factor Unverified assumption2Excessive blast radius2No resource requests or limits2Change with a delayed effect1Monitoring blind spot1No runbook1No safeguard on destructive change1No headroom1Pages not delivered1Single point of failure1Deferred upgrades1
Symptom Service outage3Scale-up failure2Pods Pending2Pod eviction2Deploys failing1Sustained high node CPU1Misrouted requests1Resources deleted1

Cases

  1. IL-0097 A cluster autoscaler version bump was incompatible with the one cluster still on Kubernetes 1.33, so no scale-up events fired and new nodes were not created2i2c · 2026

    Root cause Breaking change in an upgrade

  2. IL-0098 After 2i2c moved from ingress-nginx to nginx-ingress, a hub's Ingress took priority over cert-manager's temporary Ingress for the same host, so new hubs could not get TLS certificates2i2c · 2026

    Root cause Breaking change in an upgrade

  3. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025

    Root cause Operator error

  4. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes2i2c · 2025

    Root cause Capacity exhaustion

  5. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up2i2c · 2023

    Root cause Capacity exhaustion

  6. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)2i2c · 2022

    Root cause Cloud provider outage

  7. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub2i2c · 2020

    Root cause Capacity exhaustion