Incident Library

IL-0448 Uzabase · 2026-07-28 (published)kubernetesnetworking

Requests from Uzabase's Kubernetes Job pods to an LLM agent randomly never got responses when they ran past 15 minutes

The cluster runs kube-proxy in IPVS mode, and in the kube-proxy ConfigMap the IPVS TCP timeout was left at `tcpTimeout: 0s`, which means the default is used, and the default was 15 minutes. A TCP connection that sends and receives no packets for 15 minutes is removed from the IPVS connection table, which records which pod each connection is forwarded to. Neither the client nor the agent is told: no RST or FIN is sent, so both sides believe the connection is still open, the agent's response never reaches the client, and the client keeps waiting.

The team found this while investigating with Claude Code. It fixed the failure in the Python client by turning on TCP keepalive on the httpx transport (SO_KEEPALIVE 1, TCP_KEEPIDLE 60, TCP_KEEPINTVL 30, TCP_KEEPCNT 3, with a client timeout of 3600), so packets regularly cross the TCP connection and keep it alive.

Severity
–
Scope
Requests from Job pods to the LLM agent that took longer than about 15 minutes never received their responses
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

At Uzabase, a product team in the Speeda business uses an LLM agent to extract companies' management issues from public documents, and a client pod run as a Kubernetes Job sends the requests to the agent. The agent reviews its output many times, so a response often takes more than 15 minutes. Some share of the requests never got a response, with no clear pattern in which ones.

The client log showed that responses had not come back for some of the requests, which stayed unfinished until the forced retry the client runs after 60 minutes. The team first suspected that the agent had a problem with some requests, but the agent's log showed that it had returned its responses normally.

Comparing the client and agent logs showed that the requests that failed were the ones taking about 13 minutes or more to answer, so the team suspected a timeout somewhere on the network path inside Kubernetes.

Symptom Intermittent timeouts

Root cause

Root cause DNS or network fault

Component kube-proxy
Contributing factor Unreviewed defaultsUnknown limit
Trigger
LLM agent requests that took more than 15 minutes without packets on the TCP connection.

Detection

Client Job pods were not getting responses for some requests to the agent; the client log showed tasks that started but never finished until the 60-minute forced retry.

Remediation

Action items

–

Lessons

Application engineers also need to know the network and infrastructure. As AI applications with long response times become common, a silent cut of a long request can come from a network setting such as the IPVS TCP timeout.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, kube-proxy among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.