Cases3 cases

1 to 3 of 3

1. IL-0448 Requests from Uzabase's Kubernetes Job pods to an LLM agent randomly never got responses when they ran past 15 minutes

   Uzabase2026

   Symptoms

   At Uzabase, a product team in the Speeda business uses an LLM agent to extract companies' management issues from public documents, and a client pod run as a Kubernetes Job sends the requests to the agent. The agent reviews its output many times, so a response often takes more than 15 minutes. Some share of the requests never got a response, with no clear pattern in which ones.

   Intermittent timeouts

   Trigger

   LLM agent requests that took more than 15 minutes without packets on the TCP connection.

   Root cause

   DNS or network fault

   Component

   kube-proxy

   Contributing factor

   Unreviewed defaultsUnknown limit

   Detection

   Client Job pods were not getting responses for some requests to the agent; the client log showed tasks that started but never finished until the 60-minute forced retry.

   Remediation

   • Turned on TCP keepalive in the Python client so packets keep the connection alive

   Lessons

   Application engineers also need to know the network and infrastructure. As AI applications with long response times become common, a silent cut of a long request can come from a network setting such as the IPVS TCP timeout.

   View case
2. IL-0178 On EKS, HPA scale-in returned 502 errors because containers stopped before their pod IPs left the Service endpoints, and after blue/green switches conntrack kept sending established connections to deleted pods behind an instance-mode ALB

   Lotte E-commerce2022

   Symptoms

   LOTTE ON, Lotte E-commerce's online shopping mall of about 40 microservices on Amazon EKS, returned errors for a period of time whenever the Horizontal Pod Autoscaler (HPA) scaled pods in after load dropped. Requests reached containers that had already been deleted and failed with 502 errors.

   High error rateIntermittent timeoutsStale endpoint routing

   Trigger

   HPA scale-in deleting pods, and the deletion of Blue pods after a blue/green switch of the Service label selector.

   Root cause

   DNS or network fault

   Component

   Horizontal pod autoscalerPod terminationkube-proxyconntrack tableCloud load balancerIngress controller

   Contributing factor

   Unreviewed defaults

   Remediation

   • delayed container shutdown on pod deletion (a wait in the SIGTERM handler or a preStop sleep, capped by terminationGracePeriodSeconds) so containers stop only after the pod has left the Service endpoints
   • switched some workloads' Ingress-created ALBs from instance mode to IP mode so traffic goes straight to pod IPs without iptables and conntrack

   Action items

   • Move all workloads to ALB IP mode gradually

   Lessons

   Removing a pod from Service endpoints and stopping its containers happen at the same time in Kubernetes, so a pod can stop while traffic is still routed to it unless shutdown is delayed with a preStop hook or in the application. With an instance-mode ALB, traffic passes through node iptables and conntrack, and established connections can keep going to a pod IP after its iptables rules are gone. IP mode avoids that path, within the limit of 1000 targets per target group.

   View case
3. IL-0049 Partial DNS outage after CoreDNS scaled down and kube-proxy failed to delete stale conntrack entries

   Preply2020

   Symptoms

   Prometheus detected a high rate of 5xx responses on three services, called A, B and C in the write-up, and paged the on-call engineers through VictorOps at 22:18 EET. The 5xx rate was also visible in Kibana.

   DNS resolution failuresHigh error rateStale endpoint routing

   Trigger

   Because load in the cluster was low, the CoreDNS autoscaler reduced the number of CoreDNS pods from 3 to 2.

   Root cause

   DNS or network fault

   Component

   CoreDNSkube-proxyconntrack table

   Contributing factor

   Automation amplificationIgnored early signalNo caching

   Detection

   Prometheus alerted on a high 5xx rate on services A, B and C and paged on-call through VictorOps.

   Remediation

   • reverted the latest release of the affected service
   • a regular deploy created new nodes, the CoreDNS autoscaler added CoreDNS pods, and the conntrack table was rewritten

   Action items

   • Disable the autoscaler for CoreDNS
   • Run a DNS cache server in production
   • Monitor conntrack usage

   Lessons

   Scaling CoreDNS down is not free: if kube-proxy fails to clear conntrack entries for the removed pod, clients keep sending DNS queries to it. Known DNS weaknesses in a cluster should be prioritised before they cause an outage, and fixes that only treat consequences leave the underlying conntrack problem in place.

   View case

By root cause1