Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019
    Symptoms

    After Ravelin moved its API layer from VMs into its GKE cluster behind a Kubernetes Ingress, the integration tests occasionally received 502 errors.

    Intermittent connection failuresStale endpoint routing
    Trigger

    Pods behind the Ingress being removed by their controller, as happens during rolling deploys and scaling.

    Root cause
    Software defect
    Component
    Ingress controllerPod termination
    Contributing factor
    Unverified assumption
    Detection

    Integration tests occasionally received 502 errors.

    Lessons

    Removing a pod's endpoint and signalling the pod happen at the same time, and an ingress can take a long time to catch up. A pod behind an ingress should keep serving after SIGTERM until its grace period ends, and the grace period should be longer than the load balancer's reprogramming time.

    View case
Check your cluster for this

This case involves Ingress controller and Pod termination. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Ingress controllerPod termination
Contributing factor
Unverified assumption
Organisation
Ravelin
  1. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019