Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
    Symptoms

    Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

    Service outageNode NotReadyNode recreation loop
    Trigger

    A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

    Root cause
    Capacity exhaustion
    Component
    Jobs and CronJobsPod scheduling and placement
    Contributing factor
    Unverified assumptionAutomation amplificationExcessive blast radius
    Detection

    The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

    Remediation
    • removed the reindex Job from the scheduler
    • let the nodes auto-heal back to healthy
    Lessons

    restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

    View case
Check your cluster for this

These 6 cases involve Application database and kubelet. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Capacity exhaustion2 cases

Component
Application database1Jobs and CronJobs1Container memory limit1Pod scheduling and placement1Telemetry agent1
Contributing factor
Excessive blast radius2Unverified assumption1Automation amplification1Harmful manual action1Monitoring blind spot1No resource requests or limits1
Organisation
DFDS1Universe1
  1. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  2. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017

Cloud provider outage1 case

Component
Application databaseHelm
Contributing factor
Unverified assumptionExcessive blast radius
Organisation
Devsisters
  1. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured
    Devsisters2021

Configuration change1 case

Component
kubeletContainer memory limitPrometheus
Contributing factor
Excessive blast radiusUnreviewed defaultsMonitoring blind spotNo alert on limitNo resource requests or limits
Organisation
NU.nl
  1. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Dependency failure1 case

Component
Container runtimeCredential broker for podsService discoveryTelemetry agent
Contributing factor
Automation amplificationExcessive blast radiusUnpatched known bugSingle point of failure
Organisation
Target
  1. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019

Software defect1 case

Component
Kubernetes controller managerkubelet
Contributing factor
Excessive blast radiusNo health probes
Organisation
Oath
  1. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018