Incident Library
Applied filters FactorNo staged rolloutClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

13 to 13 of 13

  1. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
    Symptoms

    Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

    Data lossService outage
    Trigger

    A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

    Root cause
    Software defect
    Component
    Application databasePersistent volumes and stateful setsContainer memory limit
    Contributing factor
    No staged rolloutSingle point of failureUnverified assumption
    Remediation
    • restored the Redis data from the backup prepared before the rollout
    Action items
    • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
    • Reconsider the use of the Spotahome Redis Operator
    • Clean up the Redis databases and fix the application code that created the excess keys
    • Consider Redis built-in clustering and sharding
    Lessons

    Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

    View case
Check your cluster for this

These 13 cases involve Node operating system image and Application database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Capacity exhaustion2 cases

Component
In-memory cache1Application database1Kubernetes API server1Node pool1Pod scheduling and placement1
Contributing factor
No staged rollout2Retry storm2Automation amplification1No alert on limit1Ineffective rollback1Single point of failure1Unrepresentative staging1
Organisation
PostHog1Scaleway1
  1. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  2. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024

Configuration change4 cases

Component
Configuration templating2Argo CD1Cloud load balancer1Client-side DNS resolver and cache1Kubernetes API server1Telemetry agent1
Contributing factor
No staged rollout4Excessive blast radius3Unverified assumption1Automation amplification1Untested backup1Cache-masked failure1Limited debug access1Monitoring blind spot1No configuration tests1No safeguard on destructive change1Missed in review1Unrepresentative staging1
Organisation
OpenAI2Grafana Labs1Skyscanner1
  1. IL-0122 An invalid value in a configuration shared by several ChatGPT services put pods into crash loops and raised error rates
    OpenAI2025
  2. IL-0192 A Crossplane composition merge-policy change for a TLS policy update switched Kubernetes Services from LoadBalancer to ClusterIP and deleted the load balancers behind about 25% of Grafana Cloud services
    Grafana Labs2025
  3. IL-0005 A new telemetry service overloaded the Kubernetes API servers of large clusters and broke DNS-based service discovery for all OpenAI services
    OpenAI2024
  4. IL-0010 A templating change that dropped its {{ }} delimiters made Argo CD delete all 478 Skyscanner services in every region
    Skyscanner2021

Breaking change in an upgrade5 cases

Component
Node operating system image4systemd-networkd2Cilium1Cloud load balancer1Application database1Client-side DNS resolver and cache1Ingress controller1kube-proxy1kubelet1Linux kernel1Container registry and its credentials1
Contributing factor
No staged rollout5Automation amplification3Excessive blast radius1Unreviewed defaults1Change with a delayed effect1Unpatched known bug1Limited debug access1
Organisation
OpenAI2Altinity1Datadog1Microsoft Azure1
  1. IL-0120 A daily scheduled OS update restarted systemd-networkd on cloud-hosted GPU nodes, which conflicted with OpenAI's networking agent and removed all routes, cutting capacity for ChatGPT and the API
    OpenAI2025
  2. IL-0079 A GKE node auto-upgrade to an ubuntu_containerd image with a kernel regression left kube-proxy unable to apply its ip6tables rules, so load balancer health checks failed and several clusters lost all ingress traffic
    Altinity2025
  3. IL-0117 A scheduled maintenance and an ingress upgrade on OpenAI's user-facing clusters caused a networking control plane regression that briefly broke the data plane, and the services that restarted then overwhelmed a backend persistence store with a heavy first-start query
    OpenAI2024
  4. IL-0198 An automatic Ubuntu security update restarted systemd-networkd, which flushed the IP rules Cilium relied on and cut more than 60 percent of Datadog's Kubernetes nodes off the network across regions and cloud providers
    Datadog2023
  5. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022

Software defect2 cases

Component
Application database1kubelet1Node launch template1Linux kernel1Container memory limit1Node operating system image1Persistent volumes and stateful sets1Terraform1AWS VPC CNI1
Contributing factor
No staged rollout2Unverified assumption1Automation amplification1Monitoring blind spot1No health probes1Overlapping failures1Single point of failure1
Organisation
DFDS1Palark1
  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  2. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020