Incident Library

IL-0282 Amazon Web Services · 2015-09-20databaseaws

DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service

DynamoDB splits each table into partitions and spreads them over many storage servers for low latency and durability. The group of partitions assigned to one storage server is its membership, which DynamoDB's internal metadata service manages; the metadata service is replicated and runs across several datacenters. Storage servers ask the metadata service for their membership periodically, after a network disruption and at startup, and if no answer comes within a set time they retry and temporarily stop accepting requests. In the months before the event, customers had rapidly adopted Global Secondary Indexes (GSIs), which let a table be read by alternate keys and have their own partitions on the storage servers. Several GSIs on a table with many partitions could quickly double or triple its share of the membership lists. Rapid adoption by customers with very large tables raised the size of some membership lists significantly, and the processing time of some membership requests came close to the storage servers' time limit. AWS did not monitor membership size closely enough and had not given the metadata service enough capacity for these heavier requests.

At 2:19am PDT a brief network disruption sent many storage servers to the metadata service at once. Simultaneous requests for large memberships slowed processing past the limit, the affected storage servers became unavailable and retried, and the slowed metadata service then failed renewals for healthy servers too. By 2:37am the customer error rate had risen far beyond any level of the previous 3 years, and it settled at about 55%. The load blocked the administrative requests needed to add capacity, so at 5:06am AWS paused requests to the metadata service. With the retries gone, the metadata service could take administrative requests, significant capacity was added, requests were turned back on and storage servers returned to the customer request path. At 7:10am PDT error rates were low enough for most customers and dependent AWS services to resume. Amazon SQS, which refreshes cached queue information from a DynamoDB table, had significantly elevated errors after DynamoDB finished disabling traffic at 5:45am, and recovered at 7:10am with no queue data lost. EC2 Auto Scaling, which keeps its groups and launch configurations in DynamoDB, had increased API faults from 2:15am to 7:10am and then worked through a backlog of launch and termination activities until 10:52am. CloudWatch, which adds Auto Scaling group membership from a DynamoDB table to incoming EC2 metrics, had intermittent EC2 metrics from 2:35am to 5:45am and elevated PutMetricData errors from about 5:51am to 7:10am, and fully recovered at 7:29am. AWS Console logins from 5:45am to 7:10am were much slower because an API call that relied on DynamoDB, and did not need to succeed for login, had a very long timeout; a shorter timeout was already in testing but not yet rolled out.

After the main recovery a low error rate of 0.15%-0.25% remained. AWS did not see early enough that it fell disproportionately on a small number of customers, and posted to the dashboard later than it should have on Monday. The cause was a metadata partition that was still not taking its full share of traffic, and the team restored it on Monday.

Severity
–
Scope
The error rate of DynamoDB customer requests in US-East rose far beyond normal by 2:37am PDT and settled at about 55% until error rates came down at 7:10am PDT; Amazon SQS, EC2 Auto Scaling, CloudWatch and AWS Console login were also affected, and some customers saw high error rates and stuck tables into Monday
Users affected
DynamoDB customers in US-East and customers of AWS services that depend on DynamoDB
Region
US-East
Time to detect
–
Time to resolve
4h 51m

Symptoms

Early on Sunday 20 September 2015, the error rate of customer requests to Amazon DynamoDB in the US-East Region rose far beyond any level of the previous 3 years by 2:37am PDT and settled at about 55%. Some other AWS services in the region that depend on DynamoDB were affected as well.

Inside DynamoDB, the storage servers that hold the partitions of customer tables could not get their membership, the list of partitions assigned to them, from DynamoDB's internal metadata service within the time they allow for it. A storage server that cannot renew its membership in time takes itself out of the request path and retries. After a brief network disruption at 2:19am PDT, many storage servers asked for their membership at the same moment, the answers for the largest membership lists ran past the time limit, and those servers stopped taking requests.

The overloaded metadata service then also answered too slowly for storage servers that the network disruption had not touched and that were checking in on their normal schedule, so many of them became unavailable as well. Unavailable servers kept retrying and held the load high, and servers that had renewed successfully failed their next renewal and dropped out again. Downstream, Amazon SQS returned significantly elevated errors, EC2 Auto Scaling returned API faults and then lagged in launching and terminating instances until 10:52am PDT, CloudWatch had delayed and missing EC2 metrics and elevated errors on PutMetricData calls, and logins to the AWS Console became much slower.

Engineers first tried to add capacity to the metadata service, but the administrative requests needed for that failed under the same load, and several attempts did not work. After the main recovery, an error rate of 0.15%-0.25% remained. It looked too low to post on the service dashboard, but on Monday more customers opened support cases about tables stuck in the updating or deleting stage and higher than normal error rates, because that low overall rate fell heavily on a small number of customers.

Symptom High error rateDatabase overload

Root cause

Root cause Capacity exhaustion

Component Database
Contributing factor No capacity planningMonitoring blind spotRetry stormCircular dependencyExcessive blast radius
Trigger
At 2:19am PDT a brief network disruption affected a portion of DynamoDB's storage servers, and they all requested their membership data from the metadata service at the same time.

Detection

For the errors that remained after the main recovery, a growing number of customer support cases on Monday about tables stuck in the updating or deleting stage and higher than normal error rates showed that the low overall error rate was hitting some customers hard.

Remediation

Action items

Lessons

A low overall error rate can hide disproportionately high error rates for a few customers, and AWS should have posted to the dashboard sooner. A call that does not need to succeed, like the DynamoDB-dependent API call in Console login, should fail quickly instead of blocking on a very long timeout. Splitting the metadata service into many instances, each serving part of the fleet, is meant to contain the impact of software, capacity or infrastructure failures.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.