Incident Library
Applied filters Root causeunknownFactorUnverified assumptionOrganisationFyra LabsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
    Symptoms

    At around 2 AM ICT on 2023-04-09 the team saw unusual CPU use on one cluster node, which throttled every other service on it and made them fail. They presumed an upstream container runtime bug in containerd and runc and decided to fully reboot the affected nodes, taking the cluster down for maintenance on the assumption that it was not in active use.

    Sustained high node CPUService outage
    Trigger

    A full cold reboot of the nodes that showed unusual CPU use, followed by upgrades and cold reboots of the other nodes.

    Root cause
    unknown
    Component
    Multus meta-CNICalicoContainer runtime
    Contributing factor
    Harmful manual actionUnverified assumption
    Detection

    The team saw unusual CPU use on a node around 2 AM ICT and failing services on the rebooted nodes around 2:30 ICT; developers reported the Terra repositories down at 6:31 ICT.

    Remediation
    • uninstalled Multus and Calico and replaced them with the Cilium CNI
    • rolled back most cluster settings except the CNI
    Lessons

    Rebooting nodes to clear one problem can expose another; here a CNI chain could not reach the API server after a cold start. Repeating the same reboot on more nodes while the first failure was not understood spread the outage from a few nodes to the whole cluster. The team also found that Cilium's real resource use in their cluster was acceptable, contrary to the benchmark they had relied on.

    View case
Check your cluster for this

This case involves Calico and Container runtime. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

unknown1 case

Component
CalicoContainer runtimeMultus meta-CNI
Contributing factor
Unverified assumptionHarmful manual action
Organisation
Fyra Labs
  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023