IL-0055 Target · 2018-09-20 (published)kubernetes
The speaker began her career at Target, which started running Kubernetes about three years before the talk for thousands of application developers who were eager to use it but, in her words, not ready for it. She goes through the mistakes she saw there, several of which took a cluster down, and calls the failing CronJob the most common cause of a cluster outage she has seen.
A CronJob's default concurrency policy lets a new run start while earlier runs still exist. When the job fails, for example because the database it writes to is not there, each run keeps restarting and the next scheduled run adds another restarting pod instead of replacing the old one. A job scheduled every minute that fails for three days produces 4,320 restarting pods. On this cluster the pods reached about four thousand, the autoscaled cluster grew to a few hundred virtual CPUs, and the cluster went down.
Her advice is to set the CronJob concurrency policy to Replace or Forbid, and to make applications fail well: use init containers, readiness and liveness probes, and retry a missing dependency such as a database with exponential back-off for a bounded time instead of crashing and restarting against it in a tight loop.
A Kubernetes cluster went down after one CronJob kept failing and its pods piled up. Because each failed run kept restarting instead of being replaced by the next one, the pods accumulated into the thousands. The speaker's arithmetic for a job scheduled every minute that fails for three days is 4,320 restarting pods.
The cluster was autoscaled, so it grew with the pods. By the time the pods reached about four thousand, the cluster had a few hundred virtual CPUs attached, which also cost a lot of money.
Symptom Service outage
Root cause Configuration change
–
–
–
When deploying a CronJob, set its concurrency policy to Replace or Forbid; with the default, failing runs accumulate and restart together.
An application should handle a missing dependency with probes and bounded back-off rather than crash in a loop, which also keeps it from hammering a dependency that is already struggling.
IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hoursCoW Protocol · 2024same root cause · shared Node pool
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreatingUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods togetherUK Ministry of Justice · 2023same root cause · shared Node pool
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.
same root cause · shared Node pool