Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases17 cases

17 to 17 of 17

  1. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018
    Symptoms

    A Kubernetes cluster went down after one CronJob kept failing and its pods piled up. Because each failed run kept restarting instead of being replaced by the next one, the pods accumulated into the thousands. The speaker's arithmetic for a job scheduled every minute that fails for three days is 4,320 restarting pods.

    Service outage
    Trigger

    A CronJob kept failing.

    Root cause
    Configuration change
    Component
    Jobs and CronJobsNode pool
    Contributing factor
    Unreviewed defaultsAutomation amplification
    Lessons

    When deploying a CronJob, set its concurrency policy to Replace or Forbid; with the default, failing runs accumulate and restart together. An application should handle a missing dependency with probes and bounded back-off rather than crash in a loop, which also keeps it from hammering a dependency that is already struggling.

    View case
Check your cluster for this

These 17 cases involve Configuration templating and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change17 cases

Component
Configuration templating4Node pool4Cloud load balancer3Terraform3Argo CD2Kubernetes API server2kubelet2Container memory limit2Persistent volumes and stateful sets2Telemetry agent2AWS VPC CNI2Admission webhook1In-memory cache1CNI plugin1ConfigMaps and Secrets1Container runtime1Client-side DNS resolver and cache1GKE control plane1Ingress controller1Jobs and CronJobs1Node launch template1Linux kernel1Node operating system image1Pod priority and preemption1Service discovery1
Contributing factor
Automation amplification6Monitoring blind spot5Unverified assumption4Excessive blast radius4Unreviewed defaults4Missed in review4Unrepresentative staging4No configuration tests3No runbook3No staged rollout3Change with a delayed effect2Limited debug access2No health probes2No safeguard on destructive change2No headroom2Retry storm2Untested backup1Ignored early signal1Cache-masked failure1Harmful manual action1Overlapping failures1Partial rollout1Single point of failure1Unpinned version1
Organisation
DFDS3Grafana Labs3OpenAI2AdGuard VPN1CoW Protocol1DigitalOcean1InfluxData1Jetstack1Pusher1Skyscanner1Target1UK Ministry of Justice1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  4. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  5. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  6. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  7. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  8. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  9. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  10. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  11. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  12. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  13. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  14. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  15. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  16. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  17. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018