Incident Library
Applied filters Root causeConfiguration changeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases46 cases

46 to 46 of 46

  1. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018
    Symptoms

    A Kubernetes cluster went down after one CronJob kept failing and its pods piled up. Because each failed run kept restarting instead of being replaced by the next one, the pods accumulated into the thousands. The speaker's arithmetic for a job scheduled every minute that fails for three days is 4,320 restarting pods.

    Service outage
    Trigger

    A CronJob kept failing.

    Root cause
    Configuration change
    Component
    Jobs and CronJobsNode pool
    Contributing factor
    Unreviewed defaultsAutomation amplification
    Lessons

    When deploying a CronJob, set its concurrency policy to Replace or Forbid; with the default, failing runs accumulate and restart together. An application should handle a missing dependency with probes and bounded back-off rather than crash in a loop, which also keeps it from hammering a dependency that is already struggling.

    View case
Check your cluster for this

These 46 cases involve Cloud load balancer and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change46 cases

Component
Cloud load balancer9Node pool8Kubernetes API server7Configuration templating5Terraform5Ingress controller4AWS VPC CNI4etcd3kubelet3Node launch template3Container memory limit3Telemetry agent3Admission webhook2Argo CD2cert-manager2Cilium2CNI plugin2conntrack table2Container runtime2CoreDNS2Client-side DNS resolver and cache2GKE control plane2HAProxy2Helm2Linux kernel2Persistent volumes and stateful sets2Pod priority and preemption2In-memory cache1CDN and edge network1CI system1Cluster dashboard1Configuration management tool1ConfigMaps and Secrets1Coordination and key-value store1Credential broker for pods1Database1Database proxy and connection pooler1Authoritative DNS servers and records1external-dns1Horizontal pod autoscaler1HTTP and RPC client library1Jobs and CronJobs1Karpenter1Application logging library1Node root disk1Node operating system image1Node iptables rules1Pod termination1Prometheus1RBAC and service accounts1Container registry and its credentials1Service discovery1
Contributing factor
Monitoring blind spot13Unverified assumption12Excessive blast radius11Change with a delayed effect10Automation amplification8Unreviewed defaults8No configuration tests8Retry storm6Missed in review6No safeguard on destructive change5No staged rollout5Unrepresentative staging5No runbook4Ignored early signal3Limited debug access3No alert on limit3Unknown limit2Harmful manual action2No health probes2No headroom2Signal buried in noise1Untested backup1Configuration not in code1Cache-masked failure1No resource requests or limits1Overlapping failures1Partial rollout1Shared credential1Single point of failure1Unpinned version1
Organisation
DFDS5OpenAI4Grafana Labs3UK Ministry of Justice3DigitalOcean2GitLab2Wikimedia Foundation2Adevinta1AdGuard VPN1Adyen1Aqua Security1CoW Protocol1Datadog1Discord1GitHub1Google Cloud1Hans H. Kim1InfluxData1Jenkins project1Jetstack1JW Player1LIFULL1LINE Plus1MIT Open Learning1MUSINSA1NU.nl1Pusher1Python Software Foundation1Skyscanner1Spotify1Stytch1Target1
  1. IL-0179 With DEBUG logging and synchronous log output, a high-traffic service filled the 64 KB stdout pipe, so request threads blocked in write() while holding database connections and HikariCP ran dry, failing requests in 30-second bursts
    MUSINSA2026
  2. IL-0246 New pods on Adyen's big data Kubernetes nodes could not start, failing with "Cilium API client timeout exceeded"
    Adyen2026
  3. IL-0512 xpro.mit.edu returned 503 for an hour when its Kubernetes/APISIX origin certificate expired after renewal failed
    MIT Open Learning2026
  4. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
  5. IL-0200 Newly provisioned Kubernetes nodes ran an incompatible etcd version and broke cluster consensus in several GitHub production clusters, causing 5XX errors on Git operations and page loads
    GitHub2026
  6. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  7. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  8. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  9. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  10. IL-0118 A global change to Kubernetes namespace labels made the networking layer recompute metadata and overwhelmed the control planes of three of OpenAI's largest GPU clusters
    OpenAI2024
  11. IL-0130 Deployments of large models from Vertex Model Garden failed with "Model server never became ready" in most regions for 13 days after a storage provisioning configuration change
    Google Cloud2024
  12. IL-0114 An unpinned AWS VPC CNI plugin was updated automatically during a capacity expansion, leaving new pods Pending without IP addresses and stopping CoW Protocol settlements for about 3.5 hours
    CoW Protocol2024
  13. IL-0152 An incorrect configuration applied during maintenance made an internal infrastructure cluster automatically delete the control planes of 200 DOKS clusters in FRA1
    DigitalOcean2024
  14. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  15. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023
  16. IL-0459 MediaWiki on Kubernetes terminated TLS with a wrong certificate after a deployment, and 1% of requests failed
    Wikimedia Foundation2023
  17. IL-0017 AWS VPC CNI stopped allocating pod IP addresses on six nodes of the live cluster, leaving new pods in ContainerCreating
    UK Ministry of Justice2023
  18. IL-0473 Jenkins infrastructure went fully down after a pod IP range fix deleted every AKS node pool and could not recreate them
    Jenkins project2023
  19. IL-0091 Changing the worker node instance type through Terraform recycled all nodes at once and deleted users' production pods together
    UK Ministry of Justice2023
  20. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023
  21. IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local ports
    Hans H. Kim2023
  22. IL-0231 Deleting EKS managed node groups also deleted Karpenter's instance profile, and every Stytch worker node shut down
    Stytch2023
  23. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes
    DFDS2023
  24. IL-0116 A CNI configuration change incompatible with a hardware variant that exists only in production cut network connectivity in an OpenAI GPU cluster and took text-davinci-003 fully down
    OpenAI2023
  25. IL-0513 PyPI backends on Kubernetes were unreachable over HTTPS for 20 minutes when the Ingress ELB lost its HTTPS listener
    Python Software Foundation2022
  26. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx
    LINE Plus2022
  27. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022
  28. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022
  29. IL-0461 Every rollout of Datadog's metrics query service on Kubernetes raised errors that the logs blamed on DNS
    Datadog2022
  30. IL-0208 A GitLab chart bump that renamed Kubernetes Service and Deployment ports made the GCP load balancer reconfigure, and GitLab.com slowed down and returned intermittent 50x errors
    GitLab2022
  31. IL-0442 Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked
    LIFULL2021
  32. IL-0111 Switching the main Kubernetes ingress controller from Traefik v1 to v2 stopped X-Forwarded headers being passed, which broke the Okta login flow for every service using Okta and stopped DFDS online bookings for over an hour
    DFDS2021
  33. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021
  34. IL-0209 A too-high HPA CPU target kept GitLab.com web pods from scaling up as traffic moved from VMs to Kubernetes, breaching the web latency SLO for 15 minutes
    GitLab2021
  35. IL-0105 Replacing the AWS VPC CNI during a Terraform refactoring detached the ALB target groups, and every service in the Kubernetes cluster returned 503
    DFDS2021
  36. IL-0108 An Alcide admission controller applied to the production cluster during a workshop made create and update requests on Deployments time out
    DFDS2020
  37. IL-0365 A Terraform change to auto-scaling groups detached all worker nodes from both ALBs and cut off every cluster service
    DFDS2020
  38. IL-0189 A JVM service moved from EC2 to Kubernetes answered up to ten times slower because KIAM's 15-minute credentials made the AWS Java SDK fetch new credentials on every request
    Adevinta2019
  39. IL-0458 Toolforge Kubernetes API server failed to start after a restart, logging "etcd cluster is unavailable or misconfigured"
    Wikimedia Foundation2019
  40. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  41. IL-0004 A new default pod priority let a new Cortex cluster preempt every production ingester in a cascade, taking Grafana Cloud Hosted Prometheus down
    Grafana Labs2019
  42. IL-0061 A Terraform pull request merged out of order deleted a production GKE cluster, and recreating it deleted a second one
    Spotify2019
  43. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
  44. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard
    JW Player2019
  45. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019
  46. IL-0055 A failing CronJob with the default concurrency policy piled up thousands of restarting pods and took down a cluster
    Target2018