Incident Library
Applied filters Root causeunknownComponentContainer runtimeDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
    Symptoms

    At around 2 AM ICT on 2023-04-09 the team saw unusual CPU use on one cluster node, which throttled every other service on it and made them fail. They presumed an upstream container runtime bug in containerd and runc and decided to fully reboot the affected nodes, taking the cluster down for maintenance on the assumption that it was not in active use.

    Sustained high node CPUService outage
    Trigger

    A full cold reboot of the nodes that showed unusual CPU use, followed by upgrades and cold reboots of the other nodes.

    Root cause
    unknown
    Component
    Multus meta-CNICalicoContainer runtime
    Contributing factor
    Harmful manual actionUnverified assumption
    Detection

    The team saw unusual CPU use on a node around 2 AM ICT and failing services on the rebooted nodes around 2:30 ICT; developers reported the Terra repositories down at 6:31 ICT.

    Remediation
    • uninstalled Multus and Calico and replaced them with the Cilium CNI
    • rolled back most cluster settings except the CNI
    Lessons

    Rebooting nodes to clear one problem can expose another; here a CNI chain could not reach the API server after a cold start. Repeating the same reboot on more nodes while the first failure was not understood spread the outage from a few nodes to the whole cluster. The team also found that Cilium's real resource use in their cluster was acceptable, contrary to the benchmark they had relied on.

    View case
  2. IL-0507 Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodes
    UC Berkeley2017
    Symptoms

    On 10 October 2017 at 10:45 AM, a post on the course Piazza reported that two students could not start their servers on UC Berkeley's DataHub, and the /hub/admin interface could not start them either. At first it was thought the students might have run out of memory.

    Pods stuck in Terminating
    Root cause
    unknown
    Component
    Container runtime
    Contributing factor
    –
    Detection

    Students reported on the course Piazza at 10:45 AM, and later in the course Slack, that their servers would not start.

    Remediation
    • Force-deleted the stuck user pods with kubectl --namespace=prod delete pod jupyter-<name> --grace-period=0 --force
    • Cordoned node k8s-pool1-19522833-13
    • Force-deleted the stuck pods on k8s-pool2-19522833-9 and cordoned that node
    • Stopped and started k8s-pool1-19522833-13 from the Azure portal, after which Docker responded, and uncordoned it
    • Stopped and started k8s-pool2-19522833-9 from the Azure portal
    Action items
    • When several student servers fail to start or stop, check whether their pods ran on the same nodes
    • Find out how many nodes do not mount /var/lib/docker on sdb1
    • Watch for elevated counts of pods stuck in Terminating
    View case
Check your cluster for this

These 2 cases involve Container runtime and Calico. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

unknown2 cases

Component
Container runtime2Calico1Multus meta-CNI1
Contributing factor
Unverified assumption1Harmful manual action1
Organisation
Fyra Labs1UC Berkeley1
  1. IL-0087 After a cold reboot of RKE2 nodes, pods could not get IP addresses because Multus could not reach the Kubernetes API, and further reboots spread the failure to every node
    Fyra Labs2023
  2. IL-0507 Student pods on UC Berkeley DataHub got stuck in Terminating or would not start because Docker hung on two Azure nodes
    UC Berkeley2017