Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018
    Symptoms

    A Kubernetes cluster went down after one CronJob kept failing and its pods piled up. Because each failed run kept restarting instead of being replaced by the next one, the pods accumulated into the thousands. The speaker's arithmetic for a job scheduled every minute that fails for three days is 4,320 restarting pods.

    Service outage
    Trigger

    A CronJob kept failing.

    Root cause
    Configuration change
    Component
    Jobs and CronJobsNode pool
    Contributing factor
    Unreviewed defaultsAutomation amplification
    Lessons

    When deploying a CronJob, set its concurrency policy to Replace or Forbid; with the default, failing runs accumulate and restart together. An application should handle a missing dependency with probes and bounded back-off rather than crash in a loop, which also keeps it from hammering a dependency that is already struggling.

    View case
Check your cluster for this

These 8 cases involve Argo CD and Configuration templating. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change8 cases

Component
Argo CD2Configuration templating2Kubernetes API server2Node pool2Admission webhook1Cloud load balancer1Container runtime1etcd1GKE control plane1HAProxy1Helm1Jobs and CronJobs1kubelet1Node launch template1Persistent volumes and stateful sets1Pod priority and preemption1Telemetry agent1Terraform1
Contributing factor
Automation amplification8Excessive blast radius3Missed in review3Unreviewed defaults2Monitoring blind spot2No configuration tests2No runbook2No safeguard on destructive change2Untested backup1Harmful manual action1No health probes1No headroom1No staged rollout1Partial rollout1Retry storm1Unrepresentative staging1
Organisation
DFDS1DigitalOcean1GitLab1Grafana Labs1InfluxData1Jetstack1Skyscanner1Target1
  1. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  2. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  3. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  4. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  5. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  6. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  7. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  8. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018