Incident Library
Applied filters ComponentConfigMaps and SecretsDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019
    Symptoms

    During a cluster upgrade, all traffic to Pusher's products stopped. The speaker recalls that requests were failing with 500 errors.

    Service outageHigh error rate
    Trigger

    A cluster upgrade replaced nodes, so the bridge pods restarted and loaded a ConfigMap that had been changed three or four weeks earlier.

    Root cause
    Configuration change
    Component
    ConfigMaps and Secrets
    Contributing factor
    Change with a delayed effectUnverified assumption
    Action items
    • Build and run Wave so that changes to ConfigMaps and Secrets roll out new pods immediately
    Lessons

    A ConfigMap change does not restart the pods that use it, so for an application that cannot reload its configuration the change takes effect only when pods are next replaced, possibly weeks later during unrelated work. Tie configuration to the pod template so a bad change fails at the moment it is made.

    View case
Check your cluster for this

These 6 cases involve ConfigMaps and Secrets and Helm. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Breaking change in an upgrade2 cases

Component
ConfigMaps and Secrets2Admission webhook1cert-manager1Cloud load balancer1Client-side DNS resolver and cache1GKE control plane1Helm1Ingress controller1Node pool1Persistent volumes and stateful sets1
Contributing factor
Change with a delayed effect1Monitoring blind spot1No headroom1Ineffective rollback1Single point of failure1Deferred upgrades1
Organisation
PortSwigger1Wealthy1
  1. IL-0405 An expired AWS Load Balancer Controller webhook certificate left no healthy targets and took BSEE SaaS down
    PortSwigger2026
  2. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025

Faulty deployment2 cases

Component
ConfigMaps and Secrets2Helm2In-memory cache1CI system1Database1Progressive delivery controller1
Contributing factor
Signal buried in noise1Unverified assumption1Automation amplification1Partial rollout1Dependence on a warm cache1
Organisation
Gunosy1Wikimedia Foundation1
  1. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025
  2. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade
    Gunosy2023

Hardware failure1 case

Component
ConfigMaps and SecretsetcdKubesprayNode root disk
Contributing factor
Single point of failure
Organisation
Hans H. Kim
  1. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined
    Hans H. Kim2023

Configuration change1 case

Component
ConfigMaps and Secrets
Contributing factor
Unverified assumptionChange with a delayed effect
Organisation
Pusher
  1. IL-0066 Broken ConfigMap change lay dormant for weeks, then took down Pusher's edge proxy when a cluster upgrade replaced its pods
    Pusher2019