Incident Library

IL-0336 Amazon Web Services · 2021-09-02awsnetworking

AWS Direct Connect traffic into the Tokyo Region had packet loss and intermittent connectivity for about six hours

AWS Direct Connect gives customers a private connection from their own data centers to their AWS VPCs. In the Tokyo Region on 2 September 2021, a subset of network devices in one layer on the path from the Direct Connect edge locations to the Tokyo datacenter network stopped forwarding traffic correctly from 7:30 AM JST. The normal automation did not remove them. Engineers removed affected devices from service, but other devices in the layer kept failing in the same way, and resetting devices and bringing them back slowly did not hold. By 12:00 PM JST the engineers suspected a new protocol that had been introduced to shorten the network's reaction time to rare network convergence events and fiber cuts. They suspected that the protocol interacted badly with a new traffic pattern on the devices of this layer. They disabled the protocol in one Availability Zone first to confirm a lasting recovery, while preparing the same change for the whole Region. Customers reported recovery from 12:30 PM JST, and by 1:42 PM JST the devices were stable and Direct Connect worked normally.

In the following days AWS confirmed the root cause as a latent defect in the network device operating system. The operating system version that enables the new protocol, which improves the network's failover time, was first deployed to production in January 2021 and had been released gradually to all AWS Regions over 8 months without any sign of the problem. AWS had followed its usual procedure: stress tests in a dedicated lab with valid and malformed packets, then a slow production rollout on individual devices from which traffic could fail away to devices that had not been upgraded, with close monitoring. The defect needs a very specific set of packet attributes and contents, and in Tokyo customer traffic happened to produce packets matching that signature consistently. AWS states that it has no reason to suspect malicious intent. AWS disabled the protocol in Tokyo.

Severity
–
Scope
Direct Connect traffic from customer data centers to VPCs in the Tokyo Region; traffic between Availability Zones, internet connectivity to the Region, AWS VPN and Direct Connect traffic to other Regions were not affected
Users affected
Direct Connect customers connecting to the Tokyo Region
Region
Tokyo (AP-NORTHEAST-1)
Time to detect
0m
Time to resolve
6h 12m

Symptoms

From 7:30 AM JST on 2 September 2021, AWS Direct Connect customers saw intermittent connectivity problems and elevated packet loss on traffic going to their VPCs in the Tokyo (AP-NORTHEAST-1) Region. Internal AWS alarms on Direct Connect packet loss fired at the same time.

Direct Connect carries traffic from the edge locations where AWS meets customer networks to the Region through several network layers, each with many redundant devices. The alarms pointed to several devices in a single one of those layers. The devices were not forwarding traffic correctly, yet the automation that normally takes failed network devices out of service did not remove them. The automation instead saw a higher rate of failed devices than usual and called in engineers. Traffic between Availability Zones, internet traffic to the Region, AWS VPN connections, which some customers use as a backup for Direct Connect, and Direct Connect traffic to other Regions all kept working.

Engineers judged that the layer had enough redundancy and took the failing devices out of service so that healthy devices would carry the traffic. This helped only for a while: more devices then failed in the same way, and Direct Connect customers saw congestion, connectivity problems or packet loss. Resetting the failed devices and returning them to service slowly did not stop the failures, and the engineers could not keep enough healthy capacity in the layer. Engineers also looked for any recent deployment that could have set off the failure.

Symptom Packet lossIntermittent connection failures

Root cause

Root cause Software defect

Component Transit gateway or network interconnect
Contributing factor Change with a delayed effectUnrepresentative staging
Trigger
Customer traffic consistently generated packets with a very specific set of attributes and contents, and these packets set off a latent defect in the network device operating system on one layer of the Direct Connect network in Tokyo.

Detection

Internal alarms alerted AWS engineers to elevated packet loss for Direct Connect customers connecting to the Tokyo Region at 7:30 AM JST. The automation that monitors network devices also saw a higher rate of failed devices than normal and alerted engineers.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Transit gateway or network interconnect among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.