Incident Library
Applied filters SymptomProbe failuresDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases18 cases

18 to 18 of 18

  1. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
    Symptoms

    On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

    Service outageContainer crash loopProbe failuresOOMKilled
    Trigger

    A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

    Root cause
    Capacity exhaustion
    Component
    Ingress controllerContainer memory limitJupyterHub
    Contributing factor
    No resource requests or limitsHarmful manual actionNo alert on limit
    Detection

    An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

    Remediation
    • Cordoned the node that had run out of memory
    • Uncordoned the node once the proxy was working
    Action items
    • Limit the memory available to build pods
    • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
    • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
    • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod
    View case
Check your cluster for this

These 18 cases involve Ingress controller and Container memory limit. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Configuration change2 cases

Component
Container memory limit2In-memory cache1Configuration templating1Persistent volumes and stateful sets1Service discovery1
Contributing factor
Unverified assumption1Unreviewed defaults1Ignored early signal1No health probes1No headroom1Retry storm1Single point of failure1Unrepresentative staging1
Organisation
AdGuard VPN1Grafana Labs1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022

Capacity exhaustion6 cases

Component
Prometheus3Container memory limit2In-memory cache1Calico1Liveness and readiness probes1Helm1Ingress controller1Jobs and CronJobs1JupyterHub1kubelet1Linux kernel1
Contributing factor
Unknown limit3No resource requests or limits2Unreviewed defaults1Change with a delayed effect1Harmful manual action1Monitoring blind spot1No alert on limit1No capacity planning1No runbook1No headroom1Health check tied to a dependency1
Organisation
Binder Project1LINE1MUSINSA1Saifeddine Rajhi1UK Ministry of Justice1Wikimedia Foundation1
  1. IL-0581 29CM product display servers lost pods and saturated Netty event loops on the first day of the Igu Week sale
    MUSINSA2025
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
  3. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  4. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  5. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  6. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017

Dependency failure2 cases

Component
Admission webhook1Liveness and readiness probes1Istio1Node operating system image1
Contributing factor
Unreviewed defaults2Automation amplification1Excessive blast radius1Unknown limit1
Organisation
Tetrate (unnamed U.S. federal customer)1Yonghyun Kim1
  1. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  2. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025

Cloud provider outage1 case

Component
Database
Contributing factor
Health check tied to a dependency
Organisation
OpenAI
  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025

Software defect3 cases

Component
Ingress controller2Cloud load balancer1Container CPU limit1Liveness and readiness probes1kubelet1Linux kernel1
Contributing factor
Automation amplification1Excessive blast radius1Change with a delayed effect1Unpatched known bug1Unknown limit1Monitoring blind spot1No configuration tests1No headroom1
Organisation
DFDS1HubSpot1Omio1
  1. IL-0495 HubSpot's internal load balancers on Kubernetes were falsely marked unhealthy and the rest collapsed under the traffic
    HubSpot2024
  2. IL-0106 A malformed Traefik IngressRoute triggered a Traefik v2 bug once a node rollover restarted every Traefik pod, and all Kubernetes services returned 404
    DFDS2021
  3. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020

unknown2 cases

Component
etcd1Kubernetes API server1kubelet1Prometheus1
Contributing factor
Unreviewed defaults1Ignored early signal1Harmful manual action1Monitoring blind spot1Single point of failure1
Organisation
NRE Labs1UK Ministry of Justice1
  1. IL-0045 Prometheus on the live cluster restarted several times after readiness probe failures, leaving gaps in metrics
    UK Ministry of Justice2023
  2. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018

Faulty deployment1 case

Component
Service discoveryTelemetry agent
Contributing factor
Unverified assumptionExcessive blast radius
Organisation
Airbnb
  1. IL-0034 A crashing stats agent sidecar marked every pod unready and the service mesh pulled all services out of traffic
    Airbnb2019

Operator error1 case

Component
Ingress controllerkube-proxyNode pool
Contributing factor
Unverified assumptionUnreviewed defaults
Organisation
DevOps Hof
  1. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019