Cases12 cases

2 to 6 of 12

1. IL-0150 A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1

   Amazon Web Services2025

   Symptoms

   At 11:48 PM PDT on 19 October 2025, every system that reached DynamoDB in us-east-1 through the public regional endpoint started getting DNS failures and could not connect. This included customer traffic and internal AWS services that depend on DynamoDB. DynamoDB global table replicas in other regions still answered but fell behind in replication to and from us-east-1.

   DNS resolution failuresHigh error rateScale-up failureIntermittent connection failuresHigh latency

   Trigger

   One DNS Enactor, slowed by unusually high retry delays, applied a much older DNS plan to the regional DynamoDB endpoint just as another Enactor that had applied the newest plan ran its clean-up, which deleted that older plan and with it every IP address of the endpoint.

   Root cause

   Software defect

   Component

   Authoritative DNS servers and recordsApplication databaseCloud load balancer

   Contributing factor

   Automation amplificationRetry stormNo runbook

   Detection

   DNS failures to DynamoDB began immediately at 11:48 PM PDT and the engineering teams of affected AWS services were engaged at once; by 12:38 AM they had identified DynamoDB's DNS state as the source. The later NLB health check problem was detected by monitoring at 6:52 AM.

   Remediation

   • applied temporary mitigations by 1:15 AM that let some internal services reach DynamoDB and repaired key internal tooling
   • restored all DynamoDB DNS information by 2:25 AM through manual operator intervention
   • throttled incoming work and selectively restarted DWFM hosts from 4:14 AM so droplet leases could be re-established
   • reduced the load on Network Manager to bring network configuration propagation times back to normal by 10:36 AM
   • disabled automatic health check failover for NLB at 9:36 AM, and re-enabled it at 2:09 PM after EC2 recovered
   • relaxed the EC2 request throttles from 11:23 AM until all EC2 APIs and launches were normal at 1:50 PM

   Action items

   • Disable the DynamoDB DNS Planner and DNS Enactor automation worldwide
   • Fix the race condition and add protections against applying incorrect DNS plans before re-enabling the DNS automation
   • Add a velocity control to NLB that limits how much capacity a single load balancer can remove when health check failures cause availability zone failover
   • Build an additional EC2 test suite that exercises the DWFM recovery workflow at scale
   • Improve throttling in EC2 data propagation systems so incoming work is rate limited by the size of the waiting queue

   Lessons

   A single DNS record that automation can empty is enough to take a widely used regional endpoint off the network, and every service built on it fails in turn. A check made once at the start of a long operation can be stale by the time the operation writes its result. Recovery of the dependency is not recovery of the dependents: systems that lost state during the outage, such as EC2's droplet leases and network propagation backlog, can collapse under their own catch-up work, and container services that rely on new instance launches keep failing to launch and scale until that work is done.

   View case
2. IL-0131 A new kind of GenerateContent request made Vertex AI prediction API servers segfault, and once enough of them restarted at the same time the serving fleet could not carry the load

   Google Cloud2024

   Symptoms

   From 12:06 US/Pacific on 12 June 2024, services built on Vertex AI Prediction began reporting failed user requests in eight regions: us-central1, asia-southeast1, europe-west3, europe-west4, us-east1, us-west1, northamerica-northeast1 and us-east4.

   High error rateHigh latencyIntermittent timeouts

   Trigger

   Starting on 8 June 2024, a new form of user request to the GenerateContent API caused intermittent segmentation faults in the Vertex Serving Prediction API servers; on 12 June the rate of such requests increased.

   Root cause

   Software defect

   Component

   Model inference serverCloud load balancer

   Contributing factor

   Partial rolloutNo headroom

   Detection

   Internal production monitoring alerted engineers to the segfaults on 10 June 2024 at 07:35 US/Pacific, before any visible customer impact; on 12 June services built on Vertex Prediction reported failing user requests.

   Remediation

   • confirmed the new failures came from the same segfault found on 10 June
   • accelerated the rollout of the fix that was already in progress, completed at 15:59 US/Pacific

   Action items

   • Ensure early signals of server binary issues are captured in production health analysis and investigated in time
   • Ensure release validation of feature changes and updates in production

   Lessons

   A crash that only some requests trigger can stay invisible while load balancers route around the restarting servers. It becomes an outage when the share of such requests grows until too many servers are restarting at once. Here the cause was known and a fix was rolling out two days before customers were affected, but the progressive rollout had not finished when the triggering traffic increased; Google's follow-up is to investigate early signals of server binary issues in time.

   View case
3. IL-0123 A response buffer allocated anew in a loop made OpenAI's routing layer nodes hit their memory limits and fail readiness checks on a morning with more completions than any day before, causing 502 and 503 errors across all models

   OpenAI2023

   Symptoms

   Between 5:42 AM and 7:16 AM PT on 8 November 2023 a large share of requests to OpenAI failed with 502 or 503 errors. Every model and API endpoint saw significant failures.

   High error rateService outageProbe failures

   Trigger

   A morning with dramatically more completions than any previous day pushed the routing layer nodes, which already had a chronic memory problem, past their memory limits.

   Root cause

   Software defect

   Component

   Model request routingContainer memory limit

   Contributing factor

   No alert on limitNo headroom

   Remediation

   • limited incoming traffic
   • redeployed the routing service en masse
   • ramped traffic back up slowly

   Action items

   • Pre-allocate the responseBuffer and reuse it instead of allocating a new one in a loop (3X improvement in memory and CPU usage)
   • Adjust the configured memory limits to an appropriate level so the service has significant headroom
   • Add rate limit controls to shed traffic more gracefully
   • Increase the service's capacity
   • Change alerting to catch the underlying memory behaviour before it escalates into service issues
   • Configure autoscaling for the service, which was not possible before because added capacity could hurt an upstream service

   View case
4. IL-0151 A latent defect in the AWS Lambda Frontend, set off when one us-east-1 cell scaled past a size never reached before, raised errors for Lambda and for services that depend on it, including new EKS cluster provisioning

   Amazon Web Services2023

   Symptoms

   From 11:49 AM PDT on 13 June 2023, Lambda function invocations in the Northern Virginia (us-east-1) Region returned more errors and took longer. Customers that triggered functions from asynchronous or streaming event sources saw their event backlogs grow because the events were not being processed.

   High error rateHigh latencyCredentials unavailable

   Trigger

   At 11:49 AM PDT, while adding capacity for normal daily traffic growth, the Lambda Frontend fleet in one cell crossed a capacity threshold that no single cell had reached before.

   Root cause

   Software defect

   Component

   Serverless function platform

   Contributing factor

   Unknown limit

   Detection

   Engineering teams were engaged immediately and began investigating; by 12:26 PM PDT they had identified the latent defect and its effect on provisioning compute capacity.

   Remediation

   • scaled the Lambda Frontend fleet down to a size that no longer triggered the latent defect

   Action items

   • Disable the Lambda Frontend scaling activities that triggered the event
   • Fix the latent bug and deploy the fix to all Regions
   • Complete action items addressing the immediate concern with cellular scaling
   • Bound all cells to a well-tested size to avoid unexpected scaling issues

   Lessons

   A fleet that scales automatically can grow into a size that was never tested, and a defect that only appears at that size stays hidden until ordinary daily growth reaches it. AWS's answer is to cap every cell at a well-tested size. For EKS users, the event shows that provisioning new clusters in us-east-1 depended on Lambda, while clusters that were already running were not affected.

   View case
5. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes

   Google Cloud2022

   Symptoms

   From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

   High error rateResources deletedService outage

   Trigger

   A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

   Root cause

   Software defect

   Component

   Model request routing

   Contributing factor

   Automation amplificationExcessive blast radius

   Detection

   The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

   Remediation

   • restored the Vertex Prediction Endpoints that had been marked as deleted

   View case

By root cause1