Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019
    Symptoms

    After Ravelin moved its API layer from VMs into its GKE cluster behind a Kubernetes Ingress, the integration tests occasionally received 502 errors.

    Intermittent connection failuresStale endpoint routing
    Trigger

    Pods behind the Ingress being removed by their controller, as happens during rolling deploys and scaling.

    Root cause
    Software defect
    Component
    Ingress controllerPod termination
    Contributing factor
    Unverified assumption
    Detection

    Integration tests occasionally received 502 errors.

    Lessons

    Removing a pod's endpoint and signalling the pod happen at the same time, and an ingress can take a long time to catch up. A pod behind an ingress should keep serving after SIGTERM until its grace period ends, and the grace period should be longer than the load balancer's reprogramming time.

    View case
Check your cluster for this

These 6 cases involve Pod termination and Ingress controller. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause5

Configuration change1 case

Component
Coordination and key-value storeDatabase proxy and connection poolerPod termination
Contributing factor
Unknown limitRetry storm
Organisation
Discord
  1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026

Faulty deployment1 case

Component
Cloud load balancerIngress controllerPod termination
Contributing factor
Unverified assumptionUnreviewed defaultsMonitoring blind spot
Organisation
Yonghyun Kim
  1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy
    Yonghyun Kim2025

Capacity exhaustion1 case

Component
CoreDNSContainer CPU limitClient-side DNS resolver and cacheHorizontal pod autoscalerPod termination
Contributing factor
Unverified assumptionUnreviewed defaultsNo resource requests or limitsUnrepresentative staging
Organisation
MonotaRO
  1. IL-0433 Shifting API traffic from EC2 to EKS on Fargate overloaded two CoreDNS pods and caused mass name-resolution errors
    MonotaRO2023

DNS or network fault1 case

Component
Cloud load balancerconntrack tableHorizontal pod autoscalerIngress controllerkube-proxyPod termination
Contributing factor
Unreviewed defaults
Organisation
Lotte E-commerce
  1. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB
    Lotte E-commerce2022

Software defect2 cases

Component
Pod termination2Calico1Ingress controller1kubelet1
Contributing factor
Unverified assumption1
Organisation
Google Cloud1Ravelin1
  1. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  2. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic
    Ravelin2019