Incident Library
Applied filters SymptomContainer crash loopFactorNo resource requests or limitsDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
    Symptoms

    At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

    Node NotReadyContainer crash loopOOMKilledMonitoring outage
    Trigger

    Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agentContainer memory limitDatabase
    Contributing factor
    Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot
    Detection

    A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

    Remediation
    • marked the EC2 instance unhealthy so a new node started
    • cordoned and force-drained the NotReady node and terminated its EC2 instance
    • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
    • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
    • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
    • emailed the team that owns the dev container, which confirmed it was working on a fix
    Action items
    • Move grafana-agent to the monitoring node group
    • Add alerts for when bandwidth across the cluster reaches a certain threshold
    • Investigate whether memory limits can be set on the grafana-agent Helm chart
    • Investigate grafana-agent memory usage
    • Ask Grafana support whether grafana-agent should use a PVC
    • Consider pushing node logs to Loki with limited retention
    • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes
    Lessons

    Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

    View case
  2. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
    Symptoms

    One Monday morning the author opened the Prometheus dashboard to check a Kubernetes cluster for alerts and could not reach it. The Prometheus pod was in CrashLoopBackOff: kubectl get pods showed it 1/2 ready with 950 restarts over 3d20h on worker3.

    Container crash loopOOMKilledMonitoring outageProbe failures
    Trigger

    After start, the Prometheus time series count rose to 2,162,256 within five minutes and its memory use filled the 8GB host.

    Root cause
    Capacity exhaustion
    Component
    PrometheuskubeletLinux kernel
    Contributing factor
    No resource requests or limitsNo capacity planning
    Detection

    The author found it on a Monday morning when the Prometheus dashboard, opened to check the cluster for alerts, could not be reached.

    Remediation
    • removed most of the unnecessary settings from the Prometheus ConfigMap, after which the container stopped crashing
    Action items
    • Set appropriate resource requests and limits on the Prometheus pod
    • Reduce unneeded metrics by adjusting scrape targets or scrape interval
    • Estimate Prometheus memory needs, provide dedicated hosts with a node label such as deploy-topology: Prometheus, and pin the Prometheus pod to them with a nodeSelector
    Lessons

    Exit code 137 together with an OOM event in the kubelet log means the kernel killed the container, not kubelet. Under node memory pressure the kernel kills the process with the highest score, and kubelet derives each container's oom_score_adj from the pod's QoS class: a container without a memory request in a pod where another container has one ends up Burstable with a score near 1000, so it is killed first. In a large cluster, Prometheus memory needs to be estimated from the expected series count and requested explicitly.

    View case
  3. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017
    Symptoms

    On 30 November 2017 at 16:59 PST, mybinder.org operators noticed that Grafana looked wrong and that launching a new server failed with an error that a temporary user could not be created. Pods that were already running kept running, but no new ones were being created.

    Service outageContainer crash loopProbe failuresOOMKilled
    Trigger

    A build container, which ran without a memory limit, most likely used up all the memory on the node that ran the proxy pod.

    Root cause
    Capacity exhaustion
    Component
    Ingress controllerContainer memory limitJupyterHub
    Contributing factor
    No resource requests or limitsHarmful manual actionNo alert on limit
    Detection

    An operator happened to look at Grafana at 16:59 PST, saw that it looked wrong, and confirmed the error by launching a new server.

    Remediation
    • Cordoned the node that had run out of memory
    • Uncordoned the node once the proxy was working
    Action items
    • Limit the memory available to build pods
    • Change the JupyterHub configuration so user pods are not cleaned up when the jupyterhub pod restarts
    • Nominate one person to drive the response, possibly in a shared terminal, so only one person changes the cluster
    • Alert when something is wrong with the cluster, for example by counting 4xx responses on the jupyterhub pod
    View case
Check your cluster for this

These 3 cases involve Container memory limit and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion3 cases

Component
Container memory limit2Database1Ingress controller1JupyterHub1kubelet1Linux kernel1Prometheus1Telemetry agent1
Contributing factor
No resource requests or limits3Harmful manual action2Excessive blast radius1Monitoring blind spot1No alert on limit1No capacity planning1
Organisation
Binder Project1DFDS1LINE1
  1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  2. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  3. IL-0503 mybinder.org launches failed after a GKE node ran out of memory and its nginx-ingress proxy pod crash-looped
    Binder Project2017