Incident Library
Applied filters ComponentContainer memory limitSymptomService outageClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
    Symptoms

    Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

    Data lossService outage
    Trigger

    A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

    Root cause
    Software defect
    Component
    Application databasePersistent volumes and stateful setsContainer memory limit
    Contributing factor
    No staged rolloutSingle point of failureUnverified assumption
    Remediation
    • restored the Redis data from the backup prepared before the rollout
    Action items
    • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
    • Reconsider the use of the Spotahome Redis Operator
    • Clean up the Redis databases and fix the application code that created the excess keys
    • Consider Redis built-in clustering and sharding
    Lessons

    Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

    View case
Check your cluster for this

These 8 cases involve Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Capacity exhaustion3 cases

Component
Container memory limit3Node pool3CoreDNS2Client-side DNS resolver and cache1Horizontal pod autoscaler1kubelet1
Contributing factor
Unverified assumption1Automation amplification1Excessive blast radius1Unreviewed defaults1Monitoring blind spot1No caching1No resource requests or limits1No headroom1Retry storm1Single point of failure1
Organisation
2i2c1Backpackr1Buildkite1
  1. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  3. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020

Configuration change2 cases

Component
Container memory limit2In-memory cache1Configuration templating1Persistent volumes and stateful sets1Service discovery1
Contributing factor
Unverified assumption1Unreviewed defaults1Ignored early signal1No health probes1No headroom1Retry storm1Single point of failure1Unrepresentative staging1
Organisation
AdGuard VPN1Grafana Labs1
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
  2. IL-0191 A label selector accidentally added to the Loki distributor Service matched only the new pods of a rollout, which took all write traffic, were OOM-killed and stalled the rollout
    Grafana Labs2022

Software defect2 cases

Component
Container memory limit2Application database1Model request routing1Persistent volumes and stateful sets1
Contributing factor
Unverified assumption1No alert on limit1No headroom1No staged rollout1Single point of failure1
Organisation
OpenAI1Palark1
  1. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models
    OpenAI2023
  2. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020

DNS or network fault1 case

Component
Container memory limitContainer registry and its credentials
Contributing factor
Harmful manual actionOverlapping failuresSingle point of failure
Organisation
GitHub
  1. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020