Incident Library
Applied filters Root causeCapacity exhaustionComponentJobs and CronJobsFactorExcessive blast radiusClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
    Symptoms

    Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

    Service outageNode NotReadyNode recreation loop
    Trigger

    A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

    Root cause
    Capacity exhaustion
    Component
    Jobs and CronJobsPod scheduling and placement
    Contributing factor
    Unverified assumptionAutomation amplificationExcessive blast radius
    Detection

    The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

    Remediation
    • removed the reindex Job from the scheduler
    • let the nodes auto-heal back to healthy
    Lessons

    restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

    View case
Check your cluster for this

This case involves Jobs and CronJobs and Pod scheduling and placement. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Jobs and CronJobsPod scheduling and placement
Contributing factor
Unverified assumptionAutomation amplificationExcessive blast radius
Organisation
Universe
  1. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017