IL-0534 Amazon Web Services · 2017-02-28awsdatabase
Amazon S3 in US-EAST-1 could not serve GET, LIST, PUT or DELETE requests for hours while its index subsystem restarted
On the morning of February 28th, 2017 the S3 team was debugging why the S3 billing system was progressing more slowly than expected. At 9:37AM PST a team member ran a command from an established playbook to remove a small number of servers from one S3 subsystem used by billing, but one input was entered incorrectly and a larger set of servers was removed than intended. Those servers supported two other subsystems. The index subsystem manages the metadata and location of every S3 object in the region and is needed for all GET, LIST, PUT and DELETE requests; the placement subsystem allocates storage for new objects during PUT requests and needs a working index subsystem. Losing a large part of their capacity forced both subsystems into a full restart, and S3 could not serve requests until it finished. Other AWS services in US-EAST-1 that rely on S3 for storage were impaired while the S3 APIs were down.
S3 subsystems are built to lose large amounts of capacity with little customer impact, and removing capacity is a routine operation, but the index and placement subsystems had not been fully restarted in the larger regions for many years. S3 had grown a great deal, so restarting them and running the safety checks that validate the integrity of the metadata took longer than expected. The index subsystem could serve GET, LIST and DELETE requests at 12:26PM PST and was fully recovered at 1:18PM PST; the placement subsystem then recovered by 1:54PM PST, when S3 was operating normally. The tool used for the removal had allowed too much capacity to be taken away too quickly. The SHD administration console also depended on S3, so service status could not be updated on the dashboard until 11:37AM PST.
- Severity
- –
- Scope
- All Amazon S3 requests in the region, and AWS services that rely on S3 for storage, including the S3 console, new EC2 instance launches, EBS volumes needing data from S3 snapshots, AWS Lambda and the AWS Service Health Dashboard
- Users affected
- –
- Region
- Northern Virginia (US-EAST-1)
- Time to detect
- –
- Time to resolve
- 4h 17m
Symptoms
From 9:37AM PST on February 28th, 2017, Amazon S3 in the Northern Virginia (US-EAST-1) Region could not serve requests. AWS services in the region that store data in S3 failed with it, including the S3 console, new Amazon EC2 instance launches, Amazon EBS volumes that needed data from an S3 snapshot, and AWS Lambda.
S3 came back in steps. At 12:26PM PST the index subsystem had activated enough capacity for GET, LIST and DELETE requests, and those APIs were fully normal at 1:18PM PST. PUT requests also need the placement subsystem, which started recovering only after the index subsystem worked and finished at 1:54PM PST. Other AWS services that had built up a backlog of work during the disruption took longer to recover fully.
Until 11:37AM PST AWS could not update the status of individual services on the AWS Service Health Dashboard (SHD), because the SHD administration console itself depends on Amazon S3. Status went out through the AWS Twitter feed and SHD banner text instead.
Symptom Service outage
Root cause
Root cause Operator error
- Trigger
- While debugging an S3 billing system that was progressing more slowly than expected, an authorized S3 team member followed an established playbook to remove a small number of servers from one S3 subsystem used by the billing process, and entered one input of the command incorrectly.
Detection
–
Remediation
- Restarted the index subsystem, which served GET, LIST and DELETE again from 12:26PM PST
- Restarted the placement subsystem after the index subsystem worked, restoring PUT by 1:54PM PST
Action items
- Change the capacity removal tool to remove capacity more slowly
- Add safeguards that stop capacity removal when it would take a subsystem below its minimum required capacity
- Audit other operational tools for similar safety checks
- Improve the recovery time of key S3 subsystems and start the planned further partitioning of the index subsystem into cells now
- Run the SHD administration console across multiple AWS regions
Lessons
–
Sources
- https://aws.amazon.com/message/41926/ · official-status · archived copy
Related cases
IL-0378All Cloudflare R2 object writes and about 35% of reads failed when the R2 Gateway could not authenticate to storageCloudflare · 2025same root cause · shared Object storage
IL-0348Scaleway's FR-PAR region lost public internet connectivity when one 100 Gbps backbone link was saturatedScaleway · 2025same root cause · shared Object storage
IL-0218GitLab.com's primary PostgreSQL data was wiped by mistake and, with no working backup, the site was down about 18 hoursGitLab · 2017same root cause · shared Cloud block storage
IL-0538Sakura Cloud storage maintenance deleted 53 in-use customer disks, and the disks could not be restored from backupSakura Internet · 2012same root cause · shared Cloud block storage
IL-0337EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicasAmazon Web Services · 2011same root cause · shared Cloud block storage
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0534
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0534/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.