Cases1 case

1 to 1 of 1

1. IL-0186 Pods on one Kubernetes worker node could not reach external HTTP and HTTPS because storage components had added nat PREROUTING rules that redirected ports 80 and 443 to local ports

   Hans H. Kim2023

   Symptoms

   On one worker node of a Kubernetes cluster, traffic to ports 80 and 443 went missing: pods on that node could not make outbound HTTP or HTTPS requests. Infrastructure engineers and the author's teammates debugged it.

   Misrouted requests

   Trigger

   A server on which components for storage work had been installed automatically added nat REDIRECT rules for ports 80 and 443 to the node's iptables.

   Root cause

   Configuration change

   Component

   Node iptables rules

   Contributing factor

   –

   Remediation

   • had the people responsible for the storage components delete the REDIRECT rules from the node

   Lessons

   Inside Kubernetes, kube-proxy itself manages iptables rules, so the author felt that part needs to be understood in depth. When packets go missing during network tests, check each step of the path in turn, which requires knowing clearly which steps the networking goes through.

   View case

By root cause1