IL-0361 SEEK · 2021-12-10 (published)awsnetworking

Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets

SEEK had connected its AWS VPCs with peering connections, one per pair of VPCs, and was moving them onto a transit gateway that every VPC attaches to once. The cutover was planned in three steps: attach both VPCs to the transit gateway with no routes, change the routing table of the first VPC to use the transit gateway, then change the second. Between the second and third steps, packets from VPC A went to VPC B over the peering connection while packets from VPC B came back over the transit gateway. SEEK did not know before the migration that the two paths have different MTUs: a peering connection in the same region has an MTU of 9001, while a transit gateway has an MTU of 8500, drops larger packets without an ICMP 'Fragmentation Needed' reply, so path MTU discovery does not work, and clamps the MSS of every packet that crosses it. In AWS, Elastic Network Interfaces (ENIs) appear to default to an MTU of 9001.

Every mechanism that normally handles a small MTU on the path failed. TCP always sets the 'Don't Fragment' bit, so the transit gateway could not fragment the packets, and it sent no ICMP message for path MTU discovery. MSS clamping only helps when both handshake packets are clamped: the SYN from VPC A crossed the peering connection unchanged and told VPC B it could send 9001-byte packets, while only the SYN ACK from VPC B was clamped to 8500. VPC A therefore sent packets that fit, but VPC B sent 9001-byte packets that the transit gateway dropped. When the peering connection that handled search traffic for seek.com.au was migrated, large search responses stopped arriving, and the team rolled back the migration; the outage lasted 16 minutes. SEEK then confirmed the mechanism in a test setup and lowered the MTU of its EC2 instances to 8500.

Severity: –
Scope: Job searches on seek.com.au returned errors
Users affected: some users searching for jobs on seek.com.au
Region: –
Time to detect: –
Time to resolve: 16m

Symptoms

In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

Searches that return many results, such as all jobs in Melbourne, need several large packets in the response. In SEEK's later reproduction with EC2 instances, a transit gateway and a peering connection, an HTTP request from VPC A for a large file served from VPC B hung. Packet captures on both sides showed the machine in VPC B sending a 9001-byte packet that never reached VPC A, followed by endless TCP retransmission.

The transit gateway gave no sign of the drops. It does not send an ICMP 'Fragmentation Needed' reply for packets that are too large, and there are no metrics for the dropped packets. The only evidence came from the transit gateway attachment metrics: the 'packets in' count at the attachment of one VPC was roughly twice the 'packets out' count at the attachment of the other VPC.

Symptom High error rate Packet loss Intermittent timeouts

Root cause

Root cause Configuration change

Component Transit gateway or network interconnect

Contributing factor Unknown limit Unreviewed defaults Monitoring blind spot

Trigger: In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

Detection

The team noticed soon after the routing change that search on seek.com.au was not working.

Remediation

Rolled back the migration of the search traffic's peering connection to the transit gateway

Action items

Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up

Lessons

A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

Sources

https://medium.com/seek-blog/beware-of-transit-gateways-bearing-large-packets-77702c4c1b20 · blog · archived copy

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Transit gateway or network interconnect among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0361
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0361/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.