Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025sev1
    Symptoms

    The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

    Service outageContainer crash loopPersistent volume full
    Trigger

    A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

    Root cause
    Capacity exhaustion
    Component
    Persistent volumes and stateful setsIn-memory cache
    Contributing factor
    No capacity planning
    Remediation
    • backed off the Redis StatefulSet and the deployment
    • deleted the corrupted PVC and rebuilt it
    • restarted the StatefulSet and the deployment
    Action items
    • Increase the Klamm Redis cache volume to 100Mi
    • Integrate Sysdig monitoring on the namespace
    • Enable replica pods on the Redis cluster
    View case
Check your cluster for this

This case involves In-memory cache and Persistent volumes and stateful sets. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
In-memory cachePersistent volumes and stateful sets
Contributing factor
No capacity planning
Organisation
BC Government
  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025