Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases19 cases

19 to 19 of 19

  1. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged
    UC Berkeley2017
    Symptoms

    On 20 March 2017, from some time early in the day until about 13:00, some new student servers on UC Berkeley's JupyterHub stayed in Pending forever and the students got 500 errors that kept them from logging in.

    Pods PendingHigh error rate
    Trigger

    On 18 March at 16:30 the RAM per student was reduced from 2G to 1G to save resources while the nodes kept the same size of 26G RAM, and as a result the nodes hit the Google Cloud the limit on the number of disks per node.

    Root cause
    Capacity exhaustion
    Component
    Container memory limitCloud block storageNode pool
    Contributing factor
    Unknown limit
    Detection

    New student servers reported a 500 error that kept students from logging in. At 12:44 this was judged widespread, and at 12:53 `kubectl describe pod` showed the pods Pending on MaxVolumeCount.

    Remediation
    • Copied the Google Container Engine instance template and changed it to spawn smaller n1-highmem-2 nodes
    • Switched the managed instance group to the new template
    • Grew the instance group by 30 new nodes
    • Ran populate.bash so student pods would start on time on the new nodes
    • Stopped the simple autoscaler
    • Stopped and started affected user pods from the control panel
    • Restarted the hub and proxy pods
    Action items
    • Follow the status of kubernetes/kubernetes issue 24317
    • Find and fix the cause of the 'too many redirects' error in JupyterHub
    • Delete the older, larger nodes that are no longer in use
    • Alert when any pods stay Pending for a non-negligible time
    View case
Check your cluster for this

These 19 cases involve Node pool and Cluster autoscaler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Capacity exhaustion6 cases

Component
Node pool3VPC subnet or pod address range3Cluster autoscaler2AWS VPC CNI2Cloud block storage1Container runtime1Helm1Karpenter1kubelet1Container memory limit1Node root disk1Prometheus1
Contributing factor
Unreviewed defaults4Unknown limit2Monitoring blind spot2No capacity planning2Unverified assumption1Excessive blast radius1Ignored early signal1Harmful manual action1No headroom1Retry storm1Ineffective rollback1Single point of failure1
Organisation
2i2c1Adevinta1loveholidays1Neon1Saltside1UC Berkeley1
  1. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  2. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  3. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  4. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  5. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
  6. IL-0468 New JupyterHub student pods stuck Pending on MaxVolumeCount after a memory limit cut with node size unchanged
    UC Berkeley2017

Faulty deployment2 cases

Component
CI system1Cluster autoscaler1Descheduler1Node pool1
Contributing factor
Unreviewed defaults1Limited debug access1No alert on limit1No resource requests or limits1No headroom1Unrepresentative staging1
Organisation
Kubernetes project1UK Ministry of Justice1
  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  2. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019

Software defect4 cases

Component
kubelet2Calico1JupyterHub1Kubernetes controller manager1Pod scheduling and placement1Pod termination1
Contributing factor
Excessive blast radius1No alert on limit1No health probes1
Organisation
Binder Project1Google Cloud1Microsoft Azure1Oath1
  1. IL-0148 A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clusters
    Microsoft Azure2024
  2. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  3. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster
    Binder Project2018
  4. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018

Configuration change2 cases

Component
Node pool2Configuration templating1Karpenter1AWS VPC CNI1
Contributing factor
Unverified assumption1Change with a delayed effect1Ignored early signal1Overlapping failures1Shared credential1Unpinned version1
Organisation
CoW Protocol1Stytch1
  1. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  2. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023

Cloud provider outage4 cases

Component
Cluster autoscaler3Node pool3Database1GKE control plane1Horizontal pod autoscaler1Ingress controller1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Overlapping failures3Unverified assumption1Ignored early signal1Unpatched known bug1Limited debug access1Monitoring blind spot1Pages not delivered1Single point of failure1Unrepresentative staging1
Organisation
2i2c1GitLab1Grafana Labs1loveholidays1
  1. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022
  2. IL-0205 The GKE cluster autoscaler could not add nodes to GitLab.com's production cluster because Google Cloud ran out of SSD persistent disks in us-east1
    GitLab2021
  3. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019
  4. IL-0014 A GKE upgrade to a withdrawn version stalled on slow drains and a stuck node pool, then broke an ingress and confused the HPA
    loveholidays2019

Dependency failure1 case

Component
Node operating system imageNode pool
Contributing factor
Monitoring blind spot
Organisation
FREE NOW
  1. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke
    FREE NOW2019