Cases5 cases

1 to 5 of 5

1. IL-0188 Rolling updates on EKS returned 502 errors for about 30 seconds because Kubernetes deleted old pods as soon as new pods passed readiness, before the ALB had marked the new pod targets healthy

   Yonghyun Kim2025

   Symptoms

   Every rolling update of the team's microservices on Amazon EKS, behind an Application Load Balancer, was followed by intermittent 502 and 503 errors for users. A Slack message at 3 a.m. read "배포는 성공했는데, 서비스는 죽었다" (the deployment succeeded, but the service died). Every pod was Ready and Datadog was green while users saw errors.

   High error rateService outage

   Trigger

   A rolling update of a Deployment served through an ALB that registers pod IPs as targets, with no pod readiness gates.

   Root cause

   Faulty deployment

   Component

   Cloud load balancerIngress controllerPod termination

   Contributing factor

   Unverified assumptionUnreviewed defaultsMonitoring blind spot

   Detection

   A Slack message at 3 a.m. reported that the deployment had succeeded but the service was down; Datadog dashboards were green at the time.

   Remediation

   • labelled the namespace with elbv2.k8s.aws/pod-readiness-gate-inject=enabled so the AWS Load Balancer Controller injects pod readiness gates and Kubernetes waits for the ALB target to be healthy before deleting old pods

   Lessons

   A pod being Ready in Kubernetes does not mean the load balancer will send it traffic. Without a readiness gate, a rolling update can delete the old pods while the ALB still considers the new targets unhealthy, and dashboards that track pod status stay green during the errors. Making pods wait longer on shutdown reduced the errors but did not fix the start-up gap and made every deployment much slower; the readiness gate ties the pod's Ready state to the ALB's view of the target.

   View case
2. IL-0177 WebSocket connections to LINE's messaging-hub on Kubernetes dropped all at once whenever another service's change reloaded a shared HAProxy L7 load balancer, and the reconnect wave overloaded a token API and filled nf_conntrack on Nginx

   LINE+2022

   Symptoms

   The graph of WebSocket connections to messaging-hub, the messaging platform used by the Japanese food delivery service Demaecan, showed sudden deep drops even outside peak traffic. The drops kept coming back at irregular intervals. Messages still reached clients, because messaging-hub switches to Firebase Cloud Messaging when a WebSocket is gone.

   Connection resetsDatabase overloadIntermittent connection failuresHigh latency

   Trigger

   Another service that shared the same L7 load balancer deployed its own load balancer configuration in the Verda dashboard, which reloaded HAProxy; 30 minutes later, at the hard-stop-after limit, the old HAProxy process exited and closed every WebSocket connection it still held.

   Root cause

   Configuration change

   Component

   HAProxyCloud load balancerconntrack tableIngress controller

   Contributing factor

   Change with a delayed effectExcessive blast radiusRetry stormUnverified assumption

   Detection

   The developer noticed deep, sudden drops in the WebSocket connection graph that were not tied to peak traffic and kept recurring irregularly. The Nginx packet drops were found through network monitoring.

   Remediation

   • separated the Merchant system's token issuance API into its own deployment so reconnect traffic did not delay other Merchant APIs
   • scaled out Nginx with pod anti-affinity and a label selector so each pod ran on a different node, which stopped the nf_conntrack packet drops
   • raised Ingress Nginx proxy-read-timeout and proxy-send-timeout and the load balancer timeout to 3600s, which did not stop the drops
   • moved messaging-hub from the shared L7 load balancer to an L4 load balancer, which ended the WebSocket drops

   Lessons

   A reload of a shared load balancer by a different team can cut long-lived connections, and with hard-stop-after the cut comes 30 minutes later, far from the change that caused it. The author's lesson is not to narrow the scope of the search too early: trusting that the infrastructure was fine kept the investigation inside the team's own systems for days. A reconnect wave from many WebSocket clients can overload the systems that issue connection tokens and fill nf_conntrack on proxy nodes. Scaling Nginx out only helps for conntrack if the pods land on different nodes, and anti-affinity leaves extra replicas Pending once they outnumber the nodes. That messaging-hub problems can still load the connected systems remains an open problem.

   View case
3. IL-0062 Intermittent 502 errors behind a GKE Ingress because pods shut down before the load balancer stopped sending them traffic

   Ravelin2019

   Symptoms

   After Ravelin moved its API layer from VMs into its GKE cluster behind a Kubernetes Ingress, the integration tests occasionally received 502 errors.

   Intermittent connection failuresStale endpoint routing

   Trigger

   Pods behind the Ingress being removed by their controller, as happens during rolling deploys and scaling.

   Root cause

   Software defect

   Component

   Ingress controllerPod termination

   Contributing factor

   Unverified assumption

   Detection

   Integration tests occasionally received 502 errors.

   Lessons

   Removing a pod's endpoint and signalling the pod happen at the same time, and an ingress can take a long time to catch up. A pod behind an ingress should keep serving after SIGTERM until its grace period ends, and the grace period should be longer than the load balancer's reprogramming time.

   View case
4. IL-0011 HAProxy ingress piled requests onto the same pods because a Go template visited the backend map in sorted pod IP order

   Skyscanner2019

   Symptoms

   On a Sunday, a team using one of Skyscanner's Kubernetes clusters manually paged the out-of-hours Kubernetes support because their service showed elevated latency, worst at the p90 and maximum levels.

   High latencyLoad imbalanceOOMKilledIntermittent connection failures

   Trigger

   Upgrading the HAProxy ingress controller to v0.6 and going back from Service virtual IPs to individual pod IPs as backends, on the understanding that the sort-backends flag introduced in v0.5, false by default, meant backends were randomly shuffled.

   Root cause

   Software defect

   Component

   Ingress controllerHAProxyConfiguration templating

   Contributing factor

   Unrepresentative stagingUnverified assumptionIgnored early signal

   Detection

   A cluster user manually paged the out-of-hours Kubernetes support about elevated latency; Prometheus queries then showed the uneven request distribution across pods.

   Remediation

   • switched the affected service from pod IPs back to the Service virtual IP through a config change in the internal deployment tool Slingshot
   • patched the controller to keep each backend's servers in an insertion-ordered slice and changed the template to iterate over that slice
   • tested the patch in sandbox clusters and rolled it out to the production clusters

   Action items

   • Run the internal controller patch in production and return to pod IPs as the default backends
   • Raise the problem upstream with a GitHub issue and pull request
   • Test changes to cluster components against the specific behaviour and conditions that matter

   Lessons

   The team already knew that sorted backends cause hot-podding and still did not test the new flag well enough to see that it did not work; changes to cluster components need tests aimed at the exact behaviour that matters, under conditions like production. A keyword can behave differently in different parts of a language: range over a map is unordered in Go code but sorted in text/template. A small cluster component can do outsized harm to every user of the cluster and, in the end, to customers.

   View case
5. IL-0054 All ingress traffic lost while draining a GKE node pool because the load balancer only targeted new nodes without ingress-nginx pods

   DevOps Hof2019

   Symptoms

   While the old nodes of a GKE cluster were being drained to move workloads to a newly created node pool, none of the services in the cluster could be reached through ingress-nginx any more.

   Service outageProbe failures

   Trigger

   Cordoning and draining the old node pool removed those nodes from the GCP load balancer backend while the ingress-nginx pods were still running on them.

   Root cause

   Operator error

   Component

   Ingress controllerkube-proxyNode pool

   Contributing factor

   Unreviewed defaultsUnverified assumption

   Detection

   The team noticed during the drain that no service in the cluster was reachable through ingress-nginx.

   Remediation

   • deleted the nginx-ingress-controller pods still running on the old nodes so they were rescheduled on the new, ready nodes

   Lessons

   With externalTrafficPolicy: Local, the load balancer can only reach nodes that run an ingress pod, and cordoned nodes leave the load balancer backend immediately; if the ingress pods stay behind on cordoned nodes, all ingress traffic stops. Before any change that might touch traffic, look twice into the documentation.

   View case

By root cause4