Incident Library

IL-0583 Uniform Next · 2025-07-24 (published)networkingaws

Service-to-service calls through Amazon VPC Lattice failed a few times a day with read ECONNRESET

Uniform Next uses Amazon VPC Lattice for service-to-service communication between applications on ECS on Fargate, and the client application reuses HTTP connections, keeping each one open until its own idle timeout expires. AWS Support confirmed that if a client sends a packet on the same TCP connection between 60 seconds, the VPC Lattice service's idle time, and 66 seconds, when the Lattice service sends a FIN packet, Lattice returns an RST packet by design.

The team considered two fixes: keep the application's idle timeout at 60 seconds or less, or send packets periodically so a connection never stays idle for 60 seconds. Because the Lattice idle time is fixed at 60 seconds and a shorter application idle timeout costs almost no performance, the team set the application's idle timeout to 60 seconds or less, and the error stopped.

Severity
–
Scope
Some service-to-service requests through VPC Lattice failed, a few times a day
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

While watching error logs in Datadog, Uniform Next engineers found that the error Error: read ECONNRESET occurred a few times a day. A Node.js application on ECS on Fargate calls another application on ECS through a VPC Lattice service, and a connection was being reset somewhere on that path.

VPC Lattice access logs and VPC flow logs collected through CloudWatch showed the client's traffic recorded normally, but the failed requests did not appear in the target's VPC access logs or VPC flow logs. This pointed to the VPC Lattice service sending the RST packet.

tcpdump inside the ECS task, run through ECS Exec, showed the Lattice endpoint answering a data packet from the task on port 80 with a packet flagged [R.]. The connection on the same ephemeral port, 41728, had been opened with a SYN about 60 seconds before that next packet was sent.

Symptom Connection resets

Root cause

Root cause DNS or network fault

Component Cloud load balancer
Contributing factor Unknown limit
Trigger
The client sent a request on a reused TCP connection that had been idle for about 60 seconds.

Detection

Error log monitoring in Datadog showed Error: read ECONNRESET a few times a day.

Remediation

Action items

–

Lessons

An idle timeout that is too long can also waste server resources, so it needs a suitable value. Managed services remove detailed settings but impose constraints, and some of their behaviour is not written explicitly in the documentation, so they need investigation and verification before use, even though a test that fully imitates production is often hard.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.