Incident Library
Applied filters ComponentPersistent volumes and stateful setsDomainKubernetesOrganisationMagic PagesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
    Symptoms

    At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

    Service outageNode NotReadyCloud API rate limit exhausted
    Trigger

    One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

    Root cause
    Capacity exhaustion
    Component
    Cloud controller managerPersistent volumes and stateful sets
    Contributing factor
    Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook
    Detection

    The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

    Remediation
    • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
    • waited until ratelimit-remaining had built up a buffer before starting recovery
    • restarted the control plane servers
    • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
    • restarted the remaining websites by hand
    Action items
    • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
    • Get a higher Hetzner API rate limit; Hetzner support was contacted
    • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
    • Document and test a procedure for recovering Longhorn after a full cluster shutdown
    Lessons

    The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

    View case
  2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
    Symptoms

    On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

    Service outage
    Trigger

    One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

    Root cause
    Dependency failure
    Component
    DatabasePersistent volumes and stateful sets
    Contributing factor
    Harmful manual actionSingle point of failureMonitoring blind spot
    Remediation
    • Restarted the storage server to fix PVC provisioning
    • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
    • Cleaned up the corrupted cluster metadata
    • Redeployed MySQL Router
    Action items
    • Monitor the health of the storage system more closely
    • Improve the failover procedures for MySQL Router
    • Investigate and fix the status page so it shows reliable information
    Lessons

    The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

    View case
Check your cluster for this

These 2 cases involve Persistent volumes and stateful sets and Cloud controller manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Capacity exhaustion1 case

Component
Cloud controller managerPersistent volumes and stateful sets
Contributing factor
Unknown limitLimited debug accessMonitoring blind spotNo runbookRetry stormSingle point of failure
Organisation
Magic Pages
  1. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025

Dependency failure1 case

Component
DatabasePersistent volumes and stateful sets
Contributing factor
Harmful manual actionMonitoring blind spotSingle point of failure
Organisation
Magic Pages
  1. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025