Incident Library
Applied filters SymptomPacket lossOrganisationSEEKClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021
    Symptoms

    In May 2021 some users of seek.com.au got errors when they searched for jobs, for 16 minutes. SEEK was in the middle of moving the traffic between two VPCs, the one that carried searches, from a VPC peering connection to an AWS Transit Gateway, and the team noticed quickly that search was broken.

    High error ratePacket lossIntermittent timeouts
    Trigger

    In the second step of the migration, SEEK changed the routing table of the first VPC to use the transit gateway while the second VPC still routed over the peering connection, so the two directions of each connection took different paths.

    Root cause
    Configuration change
    Component
    Transit gateway or network interconnect
    Contributing factor
    Unknown limitUnreviewed defaultsMonitoring blind spot
    Detection

    The team noticed soon after the routing change that search on seek.com.au was not working.

    Remediation
    • Rolled back the migration of the search traffic's peering connection to the transit gateway
    Action items
    • Lower the MTU of EC2 instances to 8500, on Amazon Linux 2 with a script at /sbin/ifup-local that runs whenever a network interface comes up
    Lessons

    A transit gateway has a smaller MTU than a same-region peering connection, drops oversized packets silently and clamps the MSS only on packets that cross it. When a migration leaves the two directions of a TCP connection on different paths, the clamp lands on the wrong side of the handshake and large packets are lost; the fix is not to send packets larger than 8500 bytes.

    View case
Check your cluster for this

This case involves Transit gateway or network interconnect. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Transit gateway or network interconnect
Contributing factor
Unreviewed defaultsUnknown limitMonitoring blind spot
Organisation
SEEK
  1. IL-0361 Job searches on seek.com.au failed for 16 minutes as a Transit Gateway silently dropped 9001-byte packets
    SEEK2021