Incident Library
Applied filters SymptomService outageFactorAutomation amplificationClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases14 cases

14 to 14 of 14

  1. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
    Symptoms

    Shortly after a search reindex job was started, problems with the Kubernetes cluster hit Universe's reporting and search APIs and the Kubernetes cluster UI itself. The status page opened an investigation into reports of search and reporting being unavailable at 14:37 EDT on 2017-07-21.

    Service outageNode NotReadyNode recreation loop
    Trigger

    A change to the search index definition, which removed some fields from full-text indexing, was shipped, and a reindex job was started as a Kubernetes Job to apply it.

    Root cause
    Capacity exhaustion
    Component
    Jobs and CronJobsPod scheduling and placement
    Contributing factor
    Unverified assumptionAutomation amplificationExcessive blast radius
    Detection

    The team saw problems with the Kubernetes cluster shortly after starting the job, and the status page cites reports of search and reporting being unavailable.

    Remediation
    • removed the reindex Job from the scheduler
    • let the nodes auto-heal back to healthy
    Lessons

    restartPolicy: Never on the pod template does not stop a Kubernetes Job from retrying: a Job replaces failed pods and may run more than once, so a job that can exhaust a node will keep moving to new nodes.

    View case
Check your cluster for this

These 14 cases involve Application database and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Capacity exhaustion5 cases

Component
Application database3Pod scheduling and placement2Admission webhook1In-memory cache1CoreDNS1Horizontal pod autoscaler1Jobs and CronJobs1Linux kernel1Container memory limit1NAT gateway1Node pool1Container registry and its credentials1
Contributing factor
Automation amplification5Unverified assumption2Excessive blast radius2Retry storm2Single point of failure2Unknown limit1Harmful manual action1Monitoring blind spot1No headroom1
Organisation
Buildkite1Datadog1Honeycomb1Moonlight1Universe1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  3. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  4. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  5. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017

Dependency failure1 case

Component
Liveness and readiness probes
Contributing factor
Automation amplificationExcessive blast radiusUnreviewed defaults
Organisation
Yonghyun Kim
  1. IL-0187 A brief Spring Cloud Config Server downtime failed every client application's liveness probe, because the actuator health check included the config server, and Kubernetes restarted the pods in a kill loop
    Yonghyun Kim2025

Breaking change in an upgrade1 case

Component
CiliumNode operating system imagesystemd-networkd
Contributing factor
Automation amplificationUnreviewed defaultsChange with a delayed effectNo staged rollout
Organisation
Datadog
  1. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023

Configuration change6 cases

Component
Argo CD2Configuration templating2Node pool2Admission webhook1Container runtime1GKE control plane1Jobs and CronJobs1Kubernetes API server1kubelet1Node launch template1Persistent volumes and stateful sets1Pod priority and preemption1Telemetry agent1Terraform1
Contributing factor
Automation amplification6Missed in review3Excessive blast radius2Unreviewed defaults2Monitoring blind spot2No configuration tests2No runbook2Untested backup1Harmful manual action1No health probes1No safeguard on destructive change1No headroom1No staged rollout1Partial rollout1Retry storm1Unrepresentative staging1
Organisation
DFDS1Grafana Labs1InfluxData1Jetstack1Skyscanner1Target1
  1. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  2. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  3. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  4. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  5. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  6. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018

Software defect1 case

Component
Model request routing
Contributing factor
Automation amplificationExcessive blast radius
Organisation
Google Cloud
  1. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022