Incident Library

Applied filters Root causeConfiguration changeSymptomService outageClear allEvery case under the filters, with no sentence; newest first.

Suggested filters Search, and the values found in your words appear here.

Graph

Root cause
Component
Symptom
Contributing factor
Case

Cases17 cases

16 to 17 of 17

IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
Pusher2019
Symptoms
During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.
Service outage High error rate
Trigger
A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.
Root cause
Configuration change
Component
ConfigMaps and Secrets
Contributing factor
Change with a delayed effect Unverified assumption
Action items
Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
Lessons
A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.
View case
IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
Target2018
Symptoms
A Kubernetes cluster went down after one CronJob kept failing and its pods piled up. Because each failed run kept restarting instead of being replaced by the next one, the pods accumulated into the thousands. The speaker's arithmetic for a job scheduled every minute that fails for three days is 4,320 restarting pods.
Service outage
Trigger
A CronJob kept failing.
Root cause
Configuration change
Component
Jobs and CronJobs Node pool
Contributing factor
Unreviewed defaults Automation amplification
Lessons
When deploying a CronJob, set its concurrency policy to Replace or Forbid; with the default, failing runs accumulate and restart together. An application should handle a missing dependency with probes and bounded back-off rather than crash in a loop, which also keeps it from hammering a dependency that is already struggling.
View case

Check your cluster for this

These 17 cases involve Configuration templating and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change17 cases

Component: Configuration templating4 Node pool4 Cloud load balancer3 Terraform3 Argo CD2 Kubernetes API server2 kubelet2 Container memory limit2 Persistent volumes and stateful sets2 Telemetry agent2 AWS VPC CNI2 Admission webhook1 In-memory cache1 CNI plugin1 ConfigMaps and Secrets1 Container runtime1 Client-side DNS resolver and cache1 GKE control plane1 Ingress controller1 Jobs and CronJobs1 Node launch template1 Linux kernel1 Node operating system image1 Pod priority and preemption1 Service discovery1
Contributing factor: Automation amplification6 Monitoring blind spot5 Unverified assumption4 Excessive blast radius4 Unreviewed defaults4 Missed in review4 Unrepresentative staging4 No configuration tests3 No runbook3 No staged rollout3 Change with a delayed effect2 Limited debug access2 No health probes2 No safeguard on destructive change2 No headroom2 Retry storm2 Untested backup1 Ignored early signal1 Cache-masked failure1 Harmful manual action1 Overlapping failures1 Partial rollout1 Single point of failure1 Unpinned version1
Organisation: DFDS3 Grafana Labs3 OpenAI2 AdGuard VPN1 CoW Protocol1 DigitalOcean1 InfluxData1 Jetstack1 Pusher1 Skyscanner1 Target1 UK Ministry of Justice1