Incident Library
Applied filters Root causeCapacity exhaustionFactorNo runbookDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
    Symptoms

    At around 09:33 UTC one of the three control plane nodes of the Kubernetes cluster that hosts Magic Pages went offline. Looking at the node showed that it had run out of memory.

    Service outageNode NotReadyCloud API rate limit exhausted
    Trigger

    One of the three control plane nodes ran out of memory and went offline at about 09:33 UTC; the attempts to replace it with hetzner-k3s failed while the cluster's own components were exhausting the Hetzner API rate limit.

    Root cause
    Capacity exhaustion
    Component
    Cloud controller managerPersistent volumes and stateful sets
    Contributing factor
    Unknown limitRetry stormMonitoring blind spotLimited debug accessSingle point of failureNo runbook
    Detection

    The founder found the control plane node offline at about 09:33 UTC. Monitoring alerts about unreachable customer websites started around the time the replacement attempts failed, and about half an hour of investigation identified the exhausted Hetzner API rate limit.

    Remediation
    • powered off all servers of the Kubernetes cluster to stop the API requests so the Hetzner rate limit could recover
    • waited until ratelimit-remaining had built up a buffer before starting recovery
    • restarted the control plane servers
    • started the essential Longhorn components by hand so Longhorn could heal and make storage volumes available
    • restarted the remaining websites by hand
    Action items
    • Monitor the Hetzner API call rate coming from the Kubernetes cluster nodes themselves to get early warning of high usage
    • Get a higher Hetzner API rate limit; Hetzner support was contacted
    • Investigate how to reduce the cluster's reliance on a single API, including alternatives to hetzner-k3s
    • Document and test a procedure for recovering Longhorn after a full cluster shutdown
    Lessons

    The running cluster, not only the deployment tooling, consumed the cloud provider's API quota, and in a failure it consumed it fast enough that the quota never recovered. The default limit of 3600 requests per hour was not enough for failure and recovery scenarios, and API usage from inside the cluster was not monitored. A tool that hides the provider's API errors slows diagnosis. After a full, non-graceful shutdown, Longhorn needed manual intervention to come back, which lengthened the outage.

    View case
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024
    Symptoms

    At 05:01 UTC on 9 October 2024 a page fired for GatewayBackendErrorsHigh, elevated 5xx errors from page-analytics_cluster in codfw, followed by a growing number of KubernetesCalicoDown alerts for WikiKube nodes not running the calico-node pod and by MediaWikiMemcachedHighErrorRate. For about an hour uncached MediaWiki requests, such as those of logged-in users, edits and API calls, saw high latency or errors.

    High error rateHigh latencyContainer crash loopProbe failuresControl plane lag
    Trigger

    An operator used the new mwscript-k8s utility to run a batch of video transcode maintenance, the TimedMediaHandler requeueTranscodes.php script. mwscript-k8s deploys a whole helm release for every script invocation, so the batch created close to 2k Calico NetworkPolicies along with many other resources over several hours.

    Root cause
    Capacity exhaustion
    Component
    CalicoHelmJobs and CronJobs
    Contributing factor
    Change with a delayed effectNo runbook
    Detection

    Automated monitoring detected the issue and paged the on-call SRE: GatewayBackendErrorsHigh at 05:01:36 UTC, 16 KubernetesCalicoDown alerts at 05:03:32 UTC and MediaWikiMemcachedHighErrorRate at 05:04:15 UTC.

    Remediation
    • Raised the resource limits of the Calico and Typha pods and applied the change with helmfile, which redeployed all Calico pods
    Action items
    • Stop mwscript-k8s from creating so many resources
    Lessons

    The incident came close to a full outage. Automated monitoring caught it and many responders found the cause, but end users were affected and responders were unsure whether they could switch over to eqiad. The problem was noticed just before the hand-off from Americas to EMEA SRE shifts, a period with a long-known lack of coverage, and it helped that the transcodes were not queued on a Friday evening in the US.

    View case
Check your cluster for this

These 2 cases involve Calico and Cloud controller manager. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion2 cases

Component
Calico1Cloud controller manager1Helm1Jobs and CronJobs1Persistent volumes and stateful sets1
Contributing factor
No runbook2Change with a delayed effect1Unknown limit1Limited debug access1Monitoring blind spot1Retry storm1Single point of failure1
Organisation
Magic Pages1Wikimedia Foundation1
  1. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  2. IL-0232 Calico pods in the codfw WikiKube cluster hit their resource limits and crash-looped, taking nodes out one by one
    Wikimedia Foundation2024