Incident Library
Applied filters ComponentPersistent volumes and stateful setsSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017
    Symptoms

    A user reported that pods were starting up and then vanishing. When the cluster operators looked, kubectl get pods and kubectl get nodes returned two different sets of data that alternated back and forth, as if there were two opinions of the cluster's state.

    Stale or inconsistent API readsStale endpoint routingService outage
    Trigger

    etcd write latency rose far above the healthy single-digit milliseconds; the speakers say trouble starts once the database sync time crosses double digits or several hundred milliseconds.

    Root cause
    Software defect
    Component
    etcdKubernetes API serverIngress controllerPersistent volumes and stateful sets
    Contributing factor
    Unreviewed defaults
    Detection

    A user reported pods starting up and vanishing.

    Remediation
    • restored service on a replacement cluster
    • freed persistent volumes from the old cluster one by one so they could bind in the new cluster
    • updated DNS records that pointed at the old cluster's load balancer names
    Action items
    • Make the Kubernetes API server request quorum reads from etcd
    Lessons

    etcd is extremely sensitive to write latency; once the database sync time crosses double digits of milliseconds, stale reads or badly degraded performance follow. Without quorum reads, a consistent store can still hand out stale data. Documentation for a fast-moving system changes, so read it again regularly. Moving workloads to a new cluster is hard when they hold persistent volumes or when DNS records point at generated load balancer names.

    View case
Check your cluster for this

These 9 cases involve Persistent volumes and stateful sets and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Configuration change2 cases

Component
Persistent volumes and stateful sets2Argo CD1In-memory cache1Configuration templating1Container memory limit1
Contributing factor
Unverified assumption1Automation amplification1Monitoring blind spot1No health probes1No runbook1No safeguard on destructive change1No headroom1Retry storm1Missed in review1Single point of failure1
Organisation
AdGuard VPN1InfluxData1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0009 A duplicated generated name made Argo CD replace the InfluxDB Cloud production Application and delete the whole production environment
    InfluxData2022

Breaking change in an upgrade1 case

Component
ConfigMaps and SecretsClient-side DNS resolver and cacheGKE control planeNode poolPersistent volumes and stateful sets
Contributing factor
No headroomIneffective rollbackSingle point of failureDeferred upgrades
Organisation
Wealthy
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025

Operator error1 case

Component
Ingress controllerJobs and CronJobsPersistent volumes and stateful setsTerraform
Contributing factor
No runbookNo safeguard on destructive change
Organisation
2i2c
  1. IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour
    2i2c2025

Capacity exhaustion2 cases

Component
Persistent volumes and stateful sets2In-memory cache1Cloud controller manager1
Contributing factor
Unknown limit1Limited debug access1Monitoring blind spot1No capacity planning1No runbook1Retry storm1Single point of failure1
Organisation
BC Government1Magic Pages1
  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025

Software defect2 cases

Component
Persistent volumes and stateful sets2Application database1etcd1Ingress controller1Kubernetes API server1Container memory limit1
Contributing factor
Unverified assumption1Unreviewed defaults1No staged rollout1Single point of failure1
Organisation
Nordstrom1Palark1
  1. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  2. IL-0072 High etcd write latency made the API servers return stale data, and the production cluster flipped between two views of its state
    Nordstrom2017

Cloud provider outage1 case

Component
Application databaseNode root diskPersistent volumes and stateful sets
Contributing factor
Unverified assumptionUnpatched known bugOverlapping failures
Organisation
Grafana Labs
  1. IL-0193 A GCP persistent disk degradation took down a Cassandra StatefulSet whose OrderedReady policy blocked recovery, and Metrictank could not reconnect to the recreated pods, causing a 23-hour Graphite query outage
    Grafana Labs2019