IL-0536 Amazon Web Services · 2014-06-13awsdatabase
Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster
SimpleDB is a distributed datastore that replicates customer data across multiple data centers. Storage nodes hold user data, with each customer Domain replicated across a group of storage nodes, and metadata nodes record which storage nodes hold each Domain. An internal lock service, itself replicated across data centers, decides which nodes are responsible for a Domain, and every node handshakes with it periodically to confirm that it still owns its data or metadata. When power loss took down several storage nodes in one data center at once, the lock service had to de-register them from their replication groups quickly, and this sudden load raised the handshake latency for the healthy nodes. The healthy storage and metadata nodes could not finish their handshakes within the configured handshake timeout, and after several retries and timeouts the nodes removed themselves from the production cluster, so API requests failed with HTTP 500.
A storage node can only rejoin the cluster after a metadata node authorizes it, which keeps nodes with stale data out, but the metadata nodes had removed themselves for the same reason and could not authorize anyone. Engineers raised the handshake timeout values by hand and restarted a subset of metadata nodes, which then authorized the storage nodes to rejoin. At 11:16 PM PDT all APIs except CreateDomain and DeleteDomain worked normally. Those two calls are served by the metadata nodes, so the two calls were throttled until 1:30 AM to let the rest of the metadata nodes recover safely.
- Severity
- –
- Scope
- All SimpleDB API calls except a fraction of eventually consistent reads for two hours, then elevated errors for CreateDomain and DeleteDomain for a little over two more hours
- Users affected
- SimpleDB customers
- Region
- US East
- Time to detect
- –
- Time to resolve
- 4h 14m
Symptoms
From 9:16 PM to 11:16 PM PDT on June 13, Amazon SimpleDB in the US East Region was unavailable to all API calls except a fraction of eventually consistent reads, and SimpleDB API requests returned server-side errors (http response code 500).
After 11:16 PM most APIs worked again, but CreateDomain and DeleteDomain calls kept elevated error rates until 1:30 AM, because AWS deliberately throttled them while the remaining metadata nodes recovered.
Symptom Service outageHigh error rate
Root cause
Root cause Software defect
- Trigger
- Power was lost to the servers that hosted several SimpleDB storage nodes in a single data center, and those storage nodes became unavailable at the same time.
Detection
–
Remediation
- Raised the lock service handshake timeout values by hand
- Restarted a subset of metadata nodes, which could then authorize the storage nodes to rejoin the cluster
- Throttled CreateDomain and DeleteDomain calls until 1:30 AM while the remaining metadata nodes recovered
Action items
- Set a longer lock service handshake timeout in all SimpleDB Regions
- Make nodes wait and retry handshakes later with a longer timeout instead of removing themselves from the cluster after repeated timeouts
Lessons
The short handshake timeout had been chosen to detect replica failures quickly, and AWS concluded it was too low. Nodes removing themselves from the cluster right after several handshake timeouts widened the event and caused the API errors; the nodes should have waited and retried with a longer timeout.
Sources
- https://aws.amazon.com/message/65649/ · official-status · archived copy
Related cases
IL-0451Setting indexes INVISIBLE on Aurora MySQL 3.10 spiked CPU and ReadIOPS on all three readers but not on the writerDaangn · 2026same root cause · shared Database
IL-0313Decagon voice requests stalled for exactly 300 seconds behind self-managed PgBouncer and exhausted the SQLAlchemy poolDecagon · 2026same root cause · shared Database
IL-0281PostHog's Logs ClickHouse cluster silently deleted almost all its S3 data, losing US logs older than 3 daysPostHog · 2026same root cause · shared Database
IL-0310An Aurora PostgreSQL failover to an upsized reader reversed twice and backend writes failed in read-only transactionsHightouch · 2025same root cause · shared Database
IL-0150A latent race condition in DynamoDB's DNS automation left an empty DNS record for dynamodb.us-east-1.amazonaws.com, and the DynamoDB and EC2 launch failures that followed caused EKS, ECS and Fargate container launch failures and cluster scaling delays in us-east-1Amazon Web Services · 2025same root cause · shared Database
IL-0436A supremum lock left by ROLLBACK TO SAVEPOINT in a long transaction blocked inserts and filled Aurora MySQL connectionsfreee · 2025same root cause · shared Database
IL-0527OpenStreetMap.org made no replication diffs for 16 hours as PostgreSQL logical decoding failed on a memory alloc errorOpenStreetMap Foundation · 2025same root cause · shared Database
IL-0329A routine migration left PGAudit stuck holding Postgres locks, causing intermittent timeouts and a 2-minute outageincident.io · 2025same root cause · shared Database
IL-0347About 25% of Scaleway FR-PAR-1 hypervisors lost VPC connectivity when both VPC BGP route reflectors got stuckScaleway · 2024same root cause · shared Database
IL-0373Heroku Postgres Essential-tier databases in the EU region ran out of memory and refused connections, twice in four daysHeroku · 2024same root cause · shared Database
IL-0024Expensive cache-backed API calls and uncancelled database requests overloaded MongoDB and took the Hugging Face Hub down three timesHugging Face · 2024same root cause · shared Database
IL-0217Honeycomb's main MySQL database locked up under cache-miss reads and ran out of connections, a 68-minute total outageHoneycomb · 2023same root cause · shared Database
IL-0330Requests timed out for two weeks waiting on Go's database/sql connection pool, one for almost 20 secondsincident.io · 2023same root cause · shared Database
IL-0385GitHub MySQL read replicas hit a semaphore deadlock during a schema migration's table rename and kept crash-recoveringGitHub · 2021same root cause · shared Database
IL-0213Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 secondsRoblox · 2021same root cause · shared Coordination and key-value store
IL-0462Some MySQL pods on Shopify's GKE clusters took up to 30 minutes to start, slowing instance replacementShopify · 2021same root cause · shared Database
IL-0271An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hourDiscord · 2020 · sev1same root cause · shared Coordination and key-value store
IL-0196Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the dataPalark · 2020same root cause · shared Database
IL-0486Figma's PostgreSQL 9 database on RDS slowed under high write IOPS and lock contention until the site was unavailableFigma · 2020same root cause · shared Database
IL-0241A database shard could not elect a new primary, and the Stripe API failed most requests in two periods on one dayStripe · 2019same root cause · shared Database
IL-0277Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachableElastic · 2019same root cause · shared Coordination and key-value store
IL-0587Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster accessElastic · 2019same root cause · shared Coordination and key-value store
IL-0439Database access slowed after each block device and filesystem expansion because the Linux kernel dropped the page cacheCybozu · 2018same root cause · shared Database
IL-0388A Google Cloud live migration took a Discord Redis primary offline, and the failover broke API instances and then guildsDiscord · 2017same root cause · shared Database
IL-0454Baemin billing servers could not open new MariaDB connections at the start of a card-discount eventWoowa Brothers · 2016same root cause · shared Database
IL-0485Dropbox went down when an OS upgrade script reinstalled active MySQL master-replica pairs and backup restores ran slowDropbox · 2014same root cause · shared Database
IL-0589Every Kickstarter MySQL replica on Amazon RDS stopped replicating on a duplicate-key error in the backings tableKickstarter · 2013same root cause · shared Database
IL-0303GitHub's MySQL master failed over to a node with a cold buffer pool during a schema migration and GitHub.com went downGitHub · 2012same root cause · shared Database
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0536
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0536/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.