Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
    Symptoms

    Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

    Service outageNode NotReadyNode recreation loop
    Trigger

    A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

    Root cause
    Capacity exhaustion
    Component
    Jobs and CronJobsPod scheduling and placement
    Contributing factor
    Unverified assumptionAutomation amplificationExcessive blast radius
    Detection

    The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

    Remediation
    • removed the reindex Job from the scheduler
    • let the nodes auto-heal back to healthy
    Lessons

    restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

    View case
Check your cluster for this

These 6 cases involve Pod scheduling and placement and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion5 cases

Component
Pod scheduling and placement5Database2In-memory cache1CoreDNS1Horizontal pod autoscaler1Jobs and CronJobs1Karpenter1Linux kernel1Managed Kubernetes control plane and provider API1Node pool1Container registry and its credentials1
Contributing factor
Automation amplification3Single point of failure3Unverified assumption2No alert on limit2Excessive blast radius1Harmful manual action1No headroom1No staged rollout1Retry storm1Ineffective rollback1Unrepresentative staging1
Organisation
IONOS Cloud1Moonlight1PostHog1Universe1이언철1
  1. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  3. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  4. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  5. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017

Software defect1 case

Component
Pod scheduling and placement
Contributing factor
–
Organisation
Microsoft Azure
  1. IL-0148 A kube-scheduler regression in Kubernetes v1.27.14, v1.28.10 and v1.29.5 made the scheduler panic on a pod whose node affinity named a nonexistent node, so no pods were scheduled on affected AKS clusters
    Microsoft Azure2024