Incident Library
Applied filters ComponentPersistent volumes and stateful setsSymptomService outageFactorRetry stormClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025
    Symptoms

    The day began with an alert that several instances of the Redis cluster were running out of disk space. That part was fixed quickly. The outage came an hour later: right after the Redis StatefulSet was deleted and recreated at 8:10 UTC, Kubernetes restarted every Redis pod within about five minutes, and problems started to appear.

    Service outageOOMKilledProbe failuresHigh latency
    Trigger

    To make Git match the volumes that had been resized by hand, the Redis StatefulSet was deleted with orphan (non-cascading) deletion and recreated with the new storage template; Kubernetes reconciled the new spec by restarting the entire Redis cluster.

    Root cause
    Configuration change
    Component
    Persistent volumes and stateful setsIn-memory cacheContainer memory limit
    Contributing factor
    No health probesUnverified assumptionSingle point of failureRetry stormNo headroom
    Detection

    An alert about Redis instances running out of disk space started the work; the outage itself showed up as backend errors about Redis connections right after the StatefulSet was recreated.

    Remediation
    • made the Redis masters forget obsolete node identities, reset the affected replicas and rejoined them, then waited for replication to catch up
    • identified the component that was actually running out of memory, scaled the backend service and raised memory limits for that component
    • temporarily relaxed probes so backend instances took traffic immediately instead of flapping between ready and not ready
    • reverted the temporary workarounds at 10:00 UTC
    Action items
    • Add startup probes so a Redis pod is not marked ready until its dataset is loaded
    • Build an own control plane for Redis that handles restarted pods stuck to an old master address
    • Reduce dependence on the single Redis cluster: move some important data to other stores, such as a separate Redis cluster per server, and use Kafka for some backend components
    Lessons

    The disk resize was not the problem; the cleanup afterwards was. An immutable field pushes operators toward deleting and recreating the controller, and that can set off a full reconcile. Startup and readiness probes are what slow a rolling restart down to a safe pace; without them a distributed store restarts in the most fragile way. Recovery plans must expect a cluster to heal slowly, and dependent services need to degrade gracefully while it does.

    View case
  2. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
    Symptoms

    At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

    Service outageNode NotReadyCloud API rate limit exhausted
    Trigger

    One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

    Root cause
    Capacity exhaustion
    Component
    Cloud controller managerPersistent volumes and stateful sets
    Contributing factor
    Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook
    Detection

    The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

    Remediation
    • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
    • waited until ratelimit-remaining had built up a buffer before starting recovery
    • restarted the control plane servers
    • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
    • restarted the remaining websites by hand
    Action items
    • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
    • Get a higher Hetzner API rate limit; Hetzner support was contacted
    • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
    • Document and test a procedure for recovering Longhorn after a full cluster shutdown
    Lessons

    The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

    View case
Check your cluster for this

These 2 cases involve Persistent volumes and stateful sets and In-memory cache. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Configuration change1 case

Component
In-memory cacheContainer memory limitPersistent volumes and stateful sets
Contributing factor
Unverified assumptionNo health probesNo headroomRetry stormSingle point of failure
Organisation
AdGuard VPN
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025

Capacity exhaustion1 case

Component
Cloud controller managerPersistent volumes and stateful sets
Contributing factor
Unknown limitLimited debug accessMonitoring blind spotNo runbookRetry stormSingle point of failure
Organisation
Magic Pages
  1. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025