Incident Library

Organisation

Hans H. Kim

Hans H. Kim · hans-h-kim

2 cases · 2023

Root cause Configuration change1Hardware failure1
Component ConfigMaps and Secrets1etcd1Kubespray1Node root disk1Node iptables rules1
Contributing factor Single point of failure1
Symptom Node NotReady1Misrouted requests1

Cases

  1. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joinedHans H. Kim · 2023

    Root cause Hardware failure

  2. IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local portsHans H. Kim · 2023

    Root cause Configuration change