Incident Library

Organisation

Amazon Web Services

Amazon Web Services · amazon-web-services

2 cases · 2023 to 2025

Root cause Software defect2

Component Cloud load balancer1 Application database1 Authoritative DNS servers and records1 Serverless function platform1

Contributing factor Automation amplification1 Unknown limit1 No runbook1 Retry storm1

Symptom High error rate2 High latency2 Scale-up failure1 Credentials unavailable1 DNS resolution failures1 Intermittent connection failures1

Cases

IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025
Root cause Software defect
IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioningAmazon Web Services · 2023
Root cause Software defect