Incident Library
Applied filters Componentkube-proxyFactorUnverified assumptionOrganisationDevOps HofClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019
    Symptoms

    While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

    Service outageProbe failures
    Trigger

    Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

    Root cause
    Operator error
    Component
    Ingress controllerkube-proxyNode pool
    Contributing factor
    Unreviewed defaultsUnverified assumption
    Detection

    The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

    Remediation
    • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes
    Lessons

    With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

    View case
Check your cluster for this

This case involves Ingress controller and kube-proxy. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
Ingress controllerkube-proxyNode pool
Contributing factor
Unverified assumptionUnreviewed defaults
Organisation
DevOps Hof
  1. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods
    DevOps Hof2019