Incident Library

IL-0223 Amazon Web Services · 2021-12-07awsnetworking

A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs

AWS runs foundational services, including monitoring, internal DNS, authorization services and parts of the EC2 control plane, on an internal network separate from the main AWS network where most AWS services and all customer applications run. Several geographically isolated networking devices connect the two networks and provide routing and network address translation between them. At 7:30 AM PST on 7 December 2021 an automated activity that scales the capacity of one AWS service on the main network set off a large surge of connection activity from clients in the internal network that overwhelmed these devices. The delays raised latency and errors for services talking across the two networks, which caused even more connection attempts and retries and kept the devices congested. The clients have back-off behaviour meant to let systems recover from such congestion, but a latent issue in a code path that had been in production for many years kept them from backing off.

Moving internal DNS traffic off the congested paths fixed DNS at 9:28 AM PST. Isolating the largest traffic sources, disabling some heavy services and adding capacity improved the congestion significantly by 1:34 PM PST, and all network devices had recovered by 2:22 PM PST. The work went slowly because monitoring data was still not visible, the internal deployment systems were affected, and operators changed things carefully to avoid harming workloads that still worked. The main network was not affected, but control planes that use services on the internal network failed, including the EC2, ELB and Route 53 APIs, Console logins, STS, CloudWatch, VPC Endpoint access to S3 and DynamoDB, API Gateway, EventBridge, Fargate, ECS, EKS and Amazon Connect. The impaired monitoring and the Service Health Dashboard tooling's failure to fail over delayed customer updates, and support cases could not be created until 2:25 PM PST. Services built on the internal network recovered later as their backlogs cleared: STS by 4:28 PM; API Gateway largely by 4:37 PM, after its automated recycling of broken servers could use the EC2 APIs again; Amazon Connect at 4:41 PM; Fargate API errors from 5:00 PM; and EventBridge delivery latency by 6:40 PM PST. AWS disabled the scaling activities that started the event.

Severity
–
Scope
AWS services and control planes that depend on the internal network in us-east-1: EC2 APIs and instance launches, ELB, Route 53 APIs, Console logins, STS, CloudWatch, VPC Endpoints for S3 and DynamoDB, API Gateway, EventBridge, Fargate, ECS, EKS, Amazon Connect and support case creation
Users affected
–
Region
us-east-1
Time to detect
–
Time to resolve
6h 52m

Symptoms

From 7:30 AM PST on 7 December 2021, AWS services in us-east-1 that communicate between AWS's internal network and its main network saw rising latency and errors. The AWS operations teams lost real-time monitoring data at once, and the first thing found in the logs was elevated internal DNS errors.

Running EC2 instances kept working, but the EC2 APIs for launching and describing instances returned more errors and latency from 7:33 AM PST, and launches of new instances recovered only at 2:40 PM PST, so customers of Amazon RDS, EMR and Workspaces could not create new resources. ELB APIs were slow to provision load balancers and register instances, Route 53 APIs were impaired from 7:30 AM to 2:30 PM PST while existing DNS answers kept working, AWS Console logins failed, STS was slow to provide credentials through OpenID Connect, CloudWatch data was delayed and partly missing, and access to S3 and DynamoDB through VPC Endpoints was impaired. API Gateway and, in the early part of the event, EventBridge had elevated error rates, and Amazon Connect had elevated failure rates for phone calls, chat sessions and task contacts. In Fargate, ECS and EKS, tasks and pods that stopped could not be restarted, and Fargate later returned Insufficient Capacity Errors.

Because internal DNS is foundational for all services and its traffic was believed to add to the congestion, the teams first moved internal DNS traffic away from the congested paths. DNS resolution errors fully recovered at 9:28 AM PST and several services improved, but the congestion and much of the service impact remained. Operators then isolated the top traffic sources onto dedicated network devices, disabled some heavy traffic services, including EventBridge event delivery at 12:35 PM, and brought extra network capacity online.

This work went slowly because monitoring data was still not visible, because AWS's internal deployment systems run in the internal network and were affected too, and because operators changed things carefully to avoid harming the many workloads on the main network that still worked. The Service Health Dashboard tooling could not fail over to its standby region at first, and updates to it succeeded only by 8:22 AM PST. Creating support cases was impaired from 7:33 AM until 2:25 PM PST.

Congestion improved significantly by 1:34 PM PST and all network devices recovered by 2:22 PM PST. EC2 API error rates and latencies began to improve at 1:15 PM PST. STS fully recovered at 4:28 PM, API Gateway largely recovered at 4:37 PM, Fargate API error rates began to return to normal at 5:00 PM, and EventBridge event delivery latency stayed elevated until 6:40 PM PST.

Symptom Control plane lagHigh latencyHigh error rateMonitoring outageDNS resolution failures

Root cause

Root cause Software defect

Component Transit gateway or network interconnectAutomated capacity manager
Contributing factor Retry stormCircular dependency
Trigger
At 7:30 AM PST an automated activity that scales the capacity of one AWS service hosted in the main AWS network triggered unexpected behaviour from a large number of clients inside the internal network.

Detection

The congestion immediately cut off real-time monitoring data for the AWS operations teams, who turned to logs and first identified elevated internal DNS errors.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Transit gateway or network interconnect, Automated capacity manager among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.