Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0223 A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs
    Amazon Web Services2021
    Symptoms

    From 7:30 AM PST on 7 December 2021, AWS services in us-east-1 that communicate between AWS's internal network and its main network saw rising latency and errors. The AWS operations teams lost real-time monitoring data at once, and the first thing found in the logs was elevated internal DNS errors.

    Control plane lagHigh latencyHigh error rateMonitoring outageDNS resolution failures
    Trigger

    At 7:30 AM PST an automated activity that scales the capacity of one AWS service hosted in the main AWS network triggered unexpected behaviour from a large number of clients inside the internal network.

    Root cause
    Software defect
    Component
    Transit gateway or network interconnectAutomated capacity manager
    Contributing factor
    Retry stormCircular dependency
    Detection

    The congestion immediately cut off real-time monitoring data for the AWS operations teams, who turned to logs and first identified elevated internal DNS errors.

    Remediation
    • Moved internal DNS traffic away from the congested network paths
    • Isolated the top sources of traffic onto dedicated network devices
    • Disabled some heavy network traffic services, including EventBridge event delivery
    • Brought additional networking capacity online
    • Disabled the automated scaling activities that triggered the event
    Action items
    • Keep the triggering scaling activities off until all remediations are deployed
    • Fix the latent issue that kept networking clients from backing off
    • Deploy network configuration that protects the affected networking devices in a similar congestion event
    • Keep API Gateway servers healthy when the internal network is unavailable and speed up their recycling
    • Release a new Service Health Dashboard that makes service impact easier to understand
    • Run the support system actively across multiple AWS regions
    View case
Check your cluster for this

This case involves Automated capacity manager and Transit gateway or network interconnect. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect1 case

Component
Automated capacity managerTransit gateway or network interconnect
Contributing factor
Circular dependencyRetry storm
Organisation
Amazon Web Services
  1. IL-0223 A connection surge congested the devices between AWS's internal and main networks in us-east-1, breaking EC2 APIs
    Amazon Web Services2021