Incident Library
Applied filters Root causeCapacity exhaustionSymptomNode NotReadyFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
    Symptoms

    At 10:01 UTC on 2024-03-13 a Kubernetes node entered NotReady state, with CPU requests at 78% and memory requests at 82%. An alert fired in Slack at 10:03 and was picked up at 10:05. An attempt to SSH into the node at 10:13 failed because it was unresponsive; it started responding again at 10:19.

    Node NotReadyContainer crash loopOOMKilledMonitoring outage
    Trigger

    Pods of a dev container that used Hangfire to trigger jobs failed with OOM and crash-looped on a node where a Grafana Agent pod was already using a lot of memory.

    Root cause
    Capacity exhaustion
    Component
    Telemetry agentContainer memory limitApplication database
    Contributing factor
    Excessive blast radiusNo resource requests or limitsHarmful manual actionMonitoring blind spot
    Detection

    A Slack alert fired two minutes after the node went NotReady and was picked up two minutes later; the second node's NotReady at 15:03 also raised a Slack alert.

    Remediation
    • marked the EC2 instance unhealthy so a new node started
    • cordoned and force-drained the NotReady node and terminated its EC2 instance
    • removed the new LimitRange from the Grafana Agent namespace and restarted the Grafana Agent pod
    • scaled the dev container to 0 replicas, after which EC2 network traffic returned to normal
    • after the second NotReady, scaled the dev container to 0 replicas again and added a new node
    • emailed the team that owns the dev container, which confirmed it was working on a fix
    Action items
    • Move grafana-agent to the monitoring node group
    • Add alerts for when bandwidth across the cluster reaches a certain threshold
    • Investigate whether memory limits can be set on the grafana-agent Helm chart
    • Investigate grafana-agent memory usage
    • Ask Grafana support whether grafana-agent should use a PVC
    • Consider pushing node logs to Loki with limited retention
    • Create a wiki page with common metrics and log queries to check when investigating issues with Kubernetes nodes
    Lessons

    Dev workloads mixed with production let one crash-looping dev container take down nodes that production also used, and a monitoring agent outside its dedicated node group added to the memory pressure. Scaling the suspect container back up before its owners had fixed it brought the failure back on the replacement node. The team also noted that action items from earlier post-mortems need follow-up, and that knowledge of useful CloudWatch metrics had not been shared, which made the diagnosis harder.

    View case
  2. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
    Symptoms

    Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

    Service outageNode NotReadyNode recreation loop
    Trigger

    A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

    Root cause
    Capacity exhaustion
    Component
    Jobs and CronJobsPod scheduling and placement
    Contributing factor
    Unverified assumptionAutomation amplificationExcessive blast radius
    Detection

    The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

    Remediation
    • removed the reindex Job from the scheduler
    • let the nodes auto-heal back to healthy
    Lessons

    restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

    View case
Check your cluster for this

These 2 cases involve Application database and Jobs and CronJobs. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Application database1Jobs and CronJobs1Container memory limit1Pod scheduling and placement1Telemetry agent1
Contributing factor
Excessive blast radius2Unverified assumption1Automation amplification1Harmful manual action1Monitoring blind spot1No resource requests or limits1
Organisation
DFDS1Universe1
  1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  2. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017