Incident Library
Applied filters SymptomContainer crash loopDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases21 cases

21 to 21 of 21

  1. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
    Symptoms

    On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

    Service outageContainer crash loopProbe failuresOOMKilled
    Trigger

    A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

    Root cause
    Capacity exhaustion
    Component
    Ingress controllerContainer memory limitJupyterHub
    Contributing factor
    No resource requests or limitsHarmful manual actionNo alert on limit
    Detection

    An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

    Remediation
    • Cordoned the node that had run out of memory
    • Uncordoned the node once the proxy was working
    Action items
    • Limit the memory available to build pods
    • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
    • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
    • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod
    View case
Check your cluster for this

These 21 cases involve Database and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Certificate or credential expiry3 cases

Component
Admission webhook1Calico1cert-manager1etcd1Kubernetes API server1kubelet1Policy engine1Secrets store1
Contributing factor
Unreviewed defaults2Monitoring blind spot2Unverified assumption1Ignored early signal1Harmful manual action1Single point of failure1Deferred upgrades1
Organisation
ckck8031DFDS1GoCardless1
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  2. IL-0107 Kyverno stuck in a crash loop behind a webhook with failurePolicy fail blocked every change to Kubernetes Service objects for 88 minutes
    DFDS2022
  3. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

Capacity exhaustion8 cases

Component
In-memory cache3Database3Ingress controller2Container memory limit2Calico1Helm1Jobs and CronJobs1JupyterHub1kubelet1Linux kernel1Node pool1Persistent volumes and stateful sets1Pod scheduling and placement1Prometheus1Telemetry agent1
Contributing factor
Harmful manual action3No resource requests or limits3No alert on limit2No capacity planning2Retry storm2Single point of failure2Automation amplification1Excessive blast radius1Change with a delayed effect1Monitoring blind spot1No runbook1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
BC Government1Binder Project1DFDS1Honeycomb1LINE1PostHog1UK Ministry of Justice1Wikimedia Foundation1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  3. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
  4. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  5. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  6. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  7. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  8. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017

Dependency failure2 cases

Component
Admission webhook1Liveness and readiness probes1Istio1Node operating system image1
Contributing factor
Unreviewed defaults2Automation amplification1Excessive blast radius1Unknown limit1
Organisation
Tetrate (unnamed U.S. federal customer)1Yonghyun Kim1
  1. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025
  2. IL-0103 During an EKS node AMI update, Istio sidecars crash-looped because hundreds of pods restarting at once overloaded the external certificate authority behind istiod
    Tetrate (unnamed U.S. federal customer)2025

Breaking change in an upgrade2 cases

Component
ConfigMaps and Secrets1Client-side DNS resolver and cache1GKE control plane1Karpenter1Node operating system image1Node pool1Persistent volumes and stateful sets1
Contributing factor
No headroom1Ineffective rollback1Single point of failure1Deferred upgrades1Unpinned version1
Organisation
PortSwigger1Wealthy1
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  2. IL-0404 All BSEE SaaS customer pods on EKS crash-looped after Karpenter moved nodes to Bottlerocket v1.26.0, which broke Java
    PortSwigger2024

Configuration change1 case

Component
–
Contributing factor
Excessive blast radiusNo staged rollout
Organisation
OpenAI
  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025

Cloud provider outage1 case

Component
Database
Contributing factor
Health check tied to a dependency
Organisation
OpenAI
  1. IL-0119 ChatGPT web service pods crash-looped on failing health checks after a Cosmos DB failure, leaving too few healthy pods to serve the web layer
    OpenAI2025

Faulty deployment2 cases

Component
CI system1ConfigMaps and Secrets1Helm1Node pool1Progressive delivery controller1
Contributing factor
Unverified assumption1Automation amplification1Change started before off-hours1Change with a delayed effect1No alert on limit1Partial rollout1Single point of failure1
Organisation
Gunosy1Kubernetes project1
  1. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade
    Gunosy2023
  2. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019

Software defect2 cases

Component
kubelet1Telemetry agent1
Contributing factor
Unpatched known bug1Limited debug access1
Organisation
AirMap1Hyperconnect1
  1. IL-0174 After an upgrade to Fluent Bit 2.0, logging sidecars that used the Kafka output plugin kept restarting with SIGSEGV because the plugin read a corrupted record as a map without checking its type
    Hyperconnect2023
  2. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018