Incident Library
Applied filters Root causeCapacity exhaustionComponentPersistent volumes and stateful setsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025sev1
    Symptoms

    The Klamm production application was completely unavailable for about 30 minutes outside working hours. Klamm's queue workers timed out because they could not reach the Redis cache, and the services that depend on it were badly affected.

    Service outageContainer crash loopPersistent volume full
    Trigger

    A large, unexpected spike in I/O from a Klamm Laravel worker sharply increased requests to the Redis cache.

    Root cause
    Capacity exhaustion
    Component
    Persistent volumes and stateful setsIn-memory cache
    Contributing factor
    No capacity planning
    Remediation
    • backed off the Redis StatefulSet and the deployment
    • deleted the corrupted PVC and rebuilt it
    • restarted the StatefulSet and the deployment
    Action items
    • Increase the Klamm Redis cache volume to 100Mi
    • Integrate Sysdig monitoring on the namespace
    • Enable replica pods on the Redis cluster
    View case
  2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
    Symptoms

    At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

    Service outageNode NotReadyCloud API rate limit exhausted
    Trigger

    One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

    Root cause
    Capacity exhaustion
    Component
    Cloud controller managerPersistent volumes and stateful sets
    Contributing factor
    Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook
    Detection

    The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

    Remediation
    • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
    • waited until ratelimit-remaining had built up a buffer before starting recovery
    • restarted the control plane servers
    • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
    • restarted the remaining websites by hand
    Action items
    • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
    • Get a higher Hetzner API rate limit; Hetzner support was contacted
    • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
    • Document and test a procedure for recovering Longhorn after a full cluster shutdown
    Lessons

    The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

    View case
Check your cluster for this

These 2 cases involve Persistent volumes and stateful sets and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Persistent volumes and stateful sets2In-memory cache1Cloud controller manager1
Contributing factor
Unknown limit1Limited debug access1Monitoring blind spot1No capacity planning1No runbook1Retry storm1Single point of failure1
Organisation
BC Government1Magic Pages1
  1. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025