IL-0062 Ravelin · 2019-07-26 (published)kubernetesgcpnetworking

Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic

Ravelin had moved its workloads to Kubernetes on GKE and was moving its API layer off the old VMs, exposing it to the outside world through an Ingress. Integration tests then started to get occasional 502 errors.

The common picture of pod removal is that the pod's endpoint leaves the Service or load balancer first, then the pod gets its preStop hook or SIGTERM, stops listening and exits once its open connections finish. In fact the endpoint removal and the SIGTERM happen at the same time. For a normal Service the endpoint change is so fast that the gap goes unnoticed, but an ingress load balancer is reprogrammed much more slowly, so a pod can receive SIGTERM well before the ingress stops sending it new connections. A pod that does a conventional graceful shutdown refuses those connections, and clients get 5xx errors.

The author's conclusion is that the pod should mostly ignore SIGTERM: keep running and keep serving new connections, optionally hint clients to go elsewhere (for HTTP, a "Connection: close" response header), and exit only when the termination grace period ends and it is killed with SIGKILL. The grace period must be longer than the time the load balancer takes to reprogram. For third-party code whose shutdown cannot be changed, a preStop hook that sleeps for the length of the grace period keeps the pod serving.

Severity

–

Scope

occasional 502 errors on the API behind the GKE Ingress, seen in integration tests

Users affected

–

Region

–

Time to detect

–

Time to resolve

–