IL-0054 DevOps Hof · 2019-02-01 (published)kubernetesgcpnetworking
Three Kubernetes behaviours combined. First, the ingress-nginx manifest for GCP sets externalTrafficPolicy: Local on its Service by default, which keeps the client source IP but means traffic is only delivered to ingress-nginx pods on the node that received it, never forwarded to another node. Second, with that policy Kubernetes creates a GCP HTTP health check on the healthCheckNodePort's /healthz, which kube-proxy deliberately fails on every node that has no local ingress-nginx pod. Third, the Kubernetes service controller only puts nodes that are Ready and schedulable into the load balancer backend, so a cordoned node is removed as soon as the drain starts.
During the node pool migration the old nodes were cordoned and dropped out of the backend, but the ingress-nginx pods stayed on them because pod disruption budgets blocked their eviction. The backend now held only new nodes, all failing the health check because no ingress-nginx pod ran there, so every ingress on the cluster stopped receiving traffic. Deleting the leftover nginx-ingress-controller pods on the old nodes let them be scheduled on the new nodes and restored traffic.
While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.
The LoadBalancer service of ingress-nginx had changed the backend pool of the GCP load balancer so that it pointed only at the new nodes. The nginx-ingress-controller pods, however, were still running on the old nodes, because problems with pod disruption budgets had kept them from being evicted. The load balancer health checks against the new nodes all failed.
The team first expected kube-proxy to forward the traffic to the right node, since it normally sends non-local traffic on to wherever the pods run, so the question was why that did not happen here.
Symptom Service outageProbe failures
Root cause Operator error
The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.
–
With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.
IL-0095 An engineer deleted the prod namespace of the LEAP JupyterHub while believing the kubeconfig pointed at a new cluster, and the hub was down for about an hour2i2c · 2025same root cause · shared Ingress controller
IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull imagesUK Ministry of Justice · 2023same root cause · shared Node pool
IL-0115 One of OpenAI's three production AKS clusters went offline through a human error, and rebuilding it took 10½ hours because an AKS Kubernetes upgrade broke the ingress and AKS cannot be downgradedOpenAI · 2022same root cause · shared Ingress controller
IL-0089 A test cluster deletion script run with the kube context pointing at the live-1 production cluster deleted its ingress controller, monitoring, logging, cert-manager, kiam and external-dnsUK Ministry of Justice · 2020same root cause · shared Ingress controller
https://incidentlib-mcp.benchlet.com/mcpThe API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.