Incident Library
Applied filters FactorNo alert on limitDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases19 cases

19 to 19 of 19

  1. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
    Symptoms

    On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

    Service outageContainer crash loopProbe failuresOOMKilled
    Trigger

    A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

    Root cause
    Capacity exhaustion
    Component
    Ingress controllerContainer memory limitJupyterHub
    Contributing factor
    No resource requests or limitsHarmful manual actionNo alert on limit
    Detection

    An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

    Remediation
    • Cordoned the node that had run out of memory
    • Uncordoned the node once the proxy was working
    Action items
    • Limit the memory available to build pods
    • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
    • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
    • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod
    View case
Check your cluster for this

These 19 cases involve Node pool and JupyterHub. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Dependency failure1 case

Component
Cloud block storageData center power and coolingetcdInfiniBand fabric
Contributing factor
Monitoring blind spotNo alert on limitNo runbookOverlapping failuresSingle point of failure
Organisation
Nebius
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026

Configuration change3 cases

Component
Cilium1CoreDNS1kubelet1Node launch template1Container memory limit1Node root disk1Node pool1Prometheus1AWS VPC CNI1
Contributing factor
No alert on limit3Unreviewed defaults2Excessive blast radius1Change with a delayed effect1Monitoring blind spot1No configuration tests1No resource requests or limits1Retry storm1Missed in review1
Organisation
Adyen1NU.nl1UK Ministry of Justice1
  1. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"
    Adyen2026
  2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  3. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Capacity exhaustion9 cases

Component
Node pool3Database2JupyterHub2Container memory limit2Pod scheduling and placement2Prometheus2Telemetry agent2In-memory cache1conntrack table1Elasticsearch or OpenSearch1HAProxy1Ingress controller1kube-proxy1Managed Kubernetes control plane and provider API1Persistent volumes and stateful sets1Container registry and its credentials1
Contributing factor
No alert on limit9Harmful manual action2Ineffective rollback2Single point of failure2Unverified assumption1Unknown limit1Monitoring blind spot1No capacity planning1No resource requests or limits1No headroom1No staged rollout1Partial rollout1Retry storm1Unrepresentative staging1
Organisation
UK Ministry of Justice32i2c1Binder Project1Buildkite1IONOS Cloud1loveholidays1PostHog1
  1. IL-0470 Buildkite notifications were delayed after Prometheus storage filled and EKS notification workers could not autoscale
    Buildkite2026
  2. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  3. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  4. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  5. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  6. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  7. IL-0500 UToronto JupyterHub logins and server starts returned 500 errors when the hub's sqlite database disk filled up
    2i2c2022
  8. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  9. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017

Faulty deployment3 cases

Component
Node pool2CI system1Cloud load balancer1Cluster autoscaler1Descheduler1Network control plane (SDN)1
Contributing factor
No alert on limit3Unverified assumption1Excessive blast radius1Change started before off-hours1Unreviewed defaults1Change with a delayed effect1No headroom1Single point of failure1
Organisation
Google Cloud1Kubernetes project1UK Ministry of Justice1
  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  2. IL-0530 Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMs
    Google Cloud2023
  3. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019

Software defect2 cases

Component
etcd1JupyterHub1Kubernetes API server1Policy engine1
Contributing factor
No alert on limit2Limited debug access1
Organisation
Binder Project1Marcin Cuber1
  1. IL-0077 EKS cluster went read-only for over seven hours after Kyverno ephemeralreports filled the etcd database past its size limit
    Marcin Cuber2024
  2. IL-0504 mybinder.org launches failed for about five hours as undeleted user pods piled up and filled the cluster
    Binder Project2018

Certificate or credential expiry1 case

Component
Kubernetes API serverkubelet
Contributing factor
Unknown limitNo alert on limit
Organisation
dbhang
  1. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021