Cases6 cases

1 to 5 of 6

1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

   Yonghyun Kim2025

   Symptoms

   Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.

   High error rateService outage

   Trigger

   A rolling update of a Deployment served through an ALB that registers pod IPs as targets, with no pod readiness gates.

   Root cause

   Faulty deployment

   Component

   Cloud load balancerIngress controllerPod termination

   Contributing factor

   Unverified assumptionUnreviewed defaultsMonitoring blind spot

   Detection

   A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.

   Remediation

   • labelled the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled so the AWS Load Balancer Controller injects pod readiness gates and Kubernetes waits for the ALB target to be healthy before deleting old pods

   Lessons

   A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors. Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.

   View case
2. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services

   Backpackr2025

   Symptoms

   During a Lunar New Year event in January 2024, a developer posted a message in Slack suspecting an outage. While several people were checking it, the problem grew into a full outage of Backpackr's web and app services, and the company decided to stop the event.

   Service outage

   Trigger

   An unexpected traffic surge, more than three times normal, during the Lunar New Year event in January 2024.

   Root cause

   Capacity exhaustion

   Component

   conntrack tableIngress controllerIstio

   Contributing factor

   Single point of failureUnreviewed defaultsMonitoring blind spot

   Detection

   A developer reported a suspected outage in Slack; the outage became total while several people were still confirming it.

   Remediation

   • stopped the Lunar New Year event

   Action items

   • Raise net.netfilter.nf_conntrack_max on all nodes to memory bytes/16384/2, and three times that on Istio nodes, applied by the nodeOptimizer DaemonSet
   • Configure Istio mesh networking so sidecars communicate directly instead of through the ingressGateway
   • Upgrade the Istio nodes from c6.xlarge (12.5 Gbps) to n-type instances with up to 25 Gbps of network bandwidth
   • Strengthen monitoring with ENA network bandwidth and packet drops, detailed Istio ingressGateway and sidecar metrics, and CPU, memory and saturation

   Lessons

   When every request in a cluster goes through one ingress gateway node, that node's connection tracking table and network bandwidth set the limit for the whole service. The default nf_conntrack table size was too small for three times normal traffic, and once it filled, packets were dropped. After the outage Backpackr strengthened its monitoring of network bandwidth, packet drops and the Istio components.

   View case
3. IL-0053 Unauthenticated ingress-nginx admission webhook let anyone on the pod network run code and read every Secret in the cluster (IngressNightmare, CVE-2025-1974)

   Wiz2025

   Symptoms

   This is a vulnerability report, not an outage, so nothing failed visibly. A cluster is exposed if it runs Ingress NGINX Controller older than 1.12.1 or 1.11.5 with its validating admission webhook enabled. Wiz suggests `kubectl get pods --all-namespaces --selector app.kubernetes.io/name=ingress-nginx` to find it. The webhook accepts AdmissionReview requests over HTTP without authentication from any pod, and Wiz found more than 6,500 clusters exposing it to the public internet.

   Unauthorised access or a foreign workload

   Trigger

   Wiz Research reviewed the code of the Ingress NGINX admission controller and reported the first findings to Kubernetes on 31 December 2024.

   Root cause

   Software defect

   Component

   Ingress controllerAdmission webhook

   Contributing factor

   Unreviewed defaultsExcessive blast radius

   Detection

   Found by Wiz Research in a code review of the admission controller, not by an operator.

   Remediation

   • Kubernetes patched the injection flaws CVE-2025-1097, CVE-2025-1098 and CVE-2025-24514
   • Kubernetes removed NGINX configuration testing from the ingress-nginx admission controller, resolving CVE-2025-1974
   • released Ingress NGINX Controller 1.12.1 and 1.11.5

   Lessons

   Admission webhooks often run as web servers inside the cluster without authentication and with relatively high privileges, so any pod can reach them, and the flaws pair well with SSRF in an application. Wiz concludes that `nginx -t` should be considered harmful when it processes unsanitized user input. Wiz recommends upgrading, keeping the webhook off the internet and allowing only the API server to reach it with network policies. If upgrading must wait, disable the webhook (Helm: controller.admissionWebhooks.enabled=false; manual installs: delete the ValidatingWebhookConfiguration ingress-nginx-admission and remove --validating-webhook from the controller) and enable it again after the upgrade.

   View case
4. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

   Lotte E-commerce2022

   Symptoms

   LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

   High error rateIntermittent timeoutsStale endpoint routing

   Trigger

   HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

   Root cause

   DNS or network fault

   Component

   Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller

   Contributing factor

   Unreviewed defaults

   Remediation

   • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
   • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack

   Action items

   • Move all workloads to ALB IP mode gradually

   Lessons

   Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

   View case
5. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods

   DevOps Hof2019

   Symptoms

   While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

   Service outageProbe failures

   Trigger

   Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

   Root cause

   Operator error

   Component

   Ingress controllerkube-proxyNode pool

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

   Remediation

   • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes

   Lessons

   With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

   View case

By root cause5