Incident Library

IL-0455 Woowa Brothers · 2020-11-11 (published)networkingaws

Calls from an AWS service through NAT Gateway to an on-premises service returned 5xx errors every busy weekend

While testing the network path, the team found that through NAT Gateway the failure appeared only when service A did not use a connection pool. Four tests narrowed it down: with a connection pool through NAT Gateway, the public internet, IDC firewall and L4, it did not reproduce; over a VPN between AWS and the IDC, skipping NAT and the public internet, it did not reproduce; from service A in one specific zone through NAT Gateway it reproduced; and with the L4 removed from the path it did not reproduce. The problem was therefore in the IDC L4.

With DAM (Direct access mode) enabled, the L4 translates client ports, but the range did not match the 2000 to 65000 the team knew. In an L4 load test, checking the session table with the /i/slb/sess/du command showed no service ports above 15000, only 2048 to 16383, and once the TEST VIP had more than about 28,000 sessions from client overload, the Allocation fail count in the AUX Table rose. The team then mapped AWS availability zones A, B and C one to one to NAT Gateways, spread service A over the zones and ran a load test expecting the L4 to allocate about 14,000 ports per source IP (NAT Gateway IP), so the symptom should appear at about 42,000 TIME_WAIT sockets on the service B test server. The test server reached about 42,000 TIME_WAIT sockets and service A got 5xx errors, confirming the limit of about 14,000 per NAT Gateway IP, 14,000 x 3 = 42,000 with three NAT gateways.

To prevent a recurrence, the team set out to switch service A to a connection pool together with its team, and to survey the services that use NAT Gateway between AWS and the IDC and switch them to connection pools.

Severity
–
Scope
API requests from service A in AWS to on-premises service B failed with 5xx errors during weekend traffic, and user requests failed
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

5xx errors rose on API requests from service A, running in AWS, to service B, running on premises, and user requests failed. At the same time the L4 session count of service B went up.

Adding service A instances lowered the 5xx errors and the L4 session count, and service A's latency and errors fell, but they did not return to normal; replacing the service A instances brought both back to normal values. The cause stayed unclear, and the failure came back every weekend, when traffic is high. The SRE team and the development teams cleared it the same way each weekend.

Service A's EC2 instances call service B's servers over a REST API, through NAT Gateway, the public internet, the data center (IDC) firewall and an IDC L4 load balancer. In service B's packet dump a normal connection from 13.125.xx.xx (AWS NAT Gateway), source port 10141, to 58.229.xxx.xx port 443 went through the three-way handshake, data transfer and the four-way close. In a failing one the requester's SYN got only an ACK(2812932054) with a wrong sequence number instead of SYN,ACK, and the requester retransmitted the SYN 3 times. The team suspected that, as source ports were reused, TCP sockets on the service B side had not been closed properly.

The team tested three hypotheses: the source not closing TCP sockets (number of instances, timeouts, AWS zones A and C), the destination not closing them (number of servers, kernel tuning such as net.core.netdev_max_backlog=30000, net.ipv4.tcp_max_syn_backlog=65536, net.core.somaxconn=16384, net.ipv4.tcp_fin_timeout=5 and net.ipv4.tcp_max_tw_buckets="1800000"), and the network path in between (NAT Gateway performance, more NAT gateways, the IDC firewall, the IDC L4). Kernel tuning and more servers did not change the symptom.

Symptom High error rateOutbound connection exhaustion

Root cause

Root cause Capacity exhaustion

Component NAT gatewayOn-premises load balancer
Contributing factor Unknown limit
Trigger
High weekend traffic from service A, which called service B without a connection pool, through the NAT gateways.

Detection

Rising 5xx errors on service A and a rising L4 session count on service B.

Remediation

Action items

Lessons

The fix is often easy to find once the cause is analysed, but without confirming the root cause a service keeps running with the risk still in it, so Woowa Brothers treats finding the root cause as essential for preventing recurrence, alongside resolving the failure first.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, NAT gateway, On-premises load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.