Cases3 cases

1 to 3 of 3

1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

   Yonghyun Kim2025

   Symptoms

   Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.

   High error rateService outage

   Trigger

   A rolling update of a Deployment served through an ALB that registers pod IPs as targets, with no pod readiness gates.

   Root cause

   Faulty deployment

   Component

   Cloud load balancerIngress controllerPod termination

   Contributing factor

   Unverified assumptionUnreviewed defaultsMonitoring blind spot

   Detection

   A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.

   Remediation

   • labelled the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled so the AWS Load Balancer Controller injects pod readiness gates and Kubernetes waits for the ALB target to be healthy before deleting old pods

   Lessons

   A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors. Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.

   View case
2. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

   Lotte E-commerce2022

   Symptoms

   LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

   High error rateIntermittent timeoutsStale endpoint routing

   Trigger

   HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

   Root cause

   DNS or network fault

   Component

   Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller

   Contributing factor

   Unreviewed defaults

   Remediation

   • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
   • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack

   Action items

   • Move all workloads to ALB IP mode gradually

   Lessons

   Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

   View case
3. IL-0052 A cryptocurrency miner ran on every node of the development and staging clusters through a public, unauthenticated Weave Scope dashboard

   JW Player2019

   Symptoms

   Datadog alerted the DevOps team to a high normalised load average, first on the staging cluster (Day 0, 21:06 EST) and the next day on the development cluster (Day 1, 16:53 EST). The load was first put down to a legitimate service that kept scaling up and down in both environments, and because of the volume of identical alerts they were muted until things settled.

   Sustained high node CPUUnauthorised access or a foreign workload

   Trigger

   An automated crawler found the Weave Scope dashboard on a public load balancer that required no authentication. Weave Scope can open a shell in running containers, and its scope-agent containers ran privileged, as root and with the host's root filesystem mounted, so a command started there could run on the host.

   Root cause

   Configuration change

   Component

   Cluster dashboardCloud load balancer

   Contributing factor

   Ignored early signalUnreviewed defaultsExcessive blast radiusMonitoring blind spot

   Detection

   Datadog alerts on high normalised load average, muted at first as noise from a legitimate service; on Day 3 an engineer ran top on a node and found the gcc process at 100% CPU.

   Remediation

   • stopped Weave Scope and removed its deployment from all Kubernetes clusters
   • checked access logs for unauthorised access to the instances
   • confirmed the miner's outbound connections went only to a mining pool
   • isolated one affected node for later investigation
   • rotated out every node in each cluster so all affected hosts were destroyed and rebuilt

   Action items

   • Audit all Kubernetes and deployment YAML files and security groups for services that are unintentionally public or misconfigured
   • Replace load-average alerts with more meaningful monitors and tune them to signal a real anomaly
   • Research tools such as falcon or sysdig for behavioural monitoring and anomaly and intrusion detection
   • Evaluate Istio and Linkerd for observing and controlling end-to-end network traffic
   • Analyse the cost of scanning Docker images and containers for known vulnerabilities
   • Set aside time in architecture discussions to revisit older choices such as running containers as root
   • Share knowledge of large projects, such as Kubernetes version migrations, across the whole team

   Lessons

   Load average was not a useful signal for spotting a problem in the clusters, and muted noisy alerts nearly hid the miner. A load balancer that Kubernetes creates belongs to Kubernetes: a manual security group edit in the AWS console is reverted, so internal exposure must be declared in the Kubernetes YAML. A dashboard that can open shells, in front of privileged agents that mount the host's root filesystem, gives whoever reaches it root on every node.

   View case

By root cause3