Incident Library

IL-0337 Amazon Web Services · 2011-04-21awsnetworking

EBS volumes in one US East Availability Zone got stuck after a network change cut EBS nodes off from their replicas

An EBS cluster runs inside one Availability Zone and is made of EBS nodes that keep replicas of volume data. Each node uses an aggressive peer-to-peer failover: when it loses contact with a node it replicates to, it assumes that node failed and searches the cluster for space to create a new replica (re-mirroring), and access to the volume is blocked until a new primary replica is chosen. The nodes are linked by a high-bandwidth primary network and a lower-capacity replication network. A regional EBS control plane accepts API requests and acts as the authority when nodes elect primary replicas. At 12:47 AM PDT on 21 April 2011, an incorrect traffic shift during a capacity upgrade moved primary network traffic onto the replication network, so a part of one cluster had neither network working and many nodes were isolated. When the shift was rolled back, a large number of nodes began re-mirroring at the same moment, used up the cluster's free capacity and kept searching for space in a loop. This "re-mirroring storm" left about 13% of the zone's volumes stuck. Nodes did not back off aggressively enough when they found no space, and a rarely hit race condition made nodes crash when they closed many replication requests at once, which created more volumes needing re-mirroring.

The degraded cluster also hurt the whole Region. Long time-outs on create volume calls filled the EBS control plane's regional thread pool, so it failed API requests for other zones until a change at 2:40 AM PDT disabled new Create Volume requests in the affected zone, after which other EBS APIs recovered by 2:50 AM PDT. By 5:30 AM PDT, the growing number of negotiations with the control plane browned it out again, and from 8:20 AM PDT the team cut all communication between the degraded cluster and the control plane, which brought the other zones back to normal but left no EBS API access in the affected zone. At 11:30 AM PDT the team stopped servers in the degraded cluster from futilely contacting other servers for space without breaking other node-to-node traffic; the cluster stopped degrading, with 13% of volumes stuck. A control plane change at the same time ended the elevated errors on new EBS-backed instance launches in other zones by noon. By 12:04 PM PDT the outage was contained to the one zone.

Recovery then meant adding storage capacity so stuck volumes could find room. EBS does not reuse a failed node until all its replicas are re-mirrored, so the team had to move spare servers physically from across the US East Region into the cluster, and the changes that had cut node-to-node communication to stabilise the cluster made it hard to bring the new servers in. From about 02:00 AM PDT on 22 April new capacity worked through the backlog, and by 12:30 PM PDT all but about 2.2% of the zone's volumes were restored. To reopen control plane access without overwhelming it, the team built a separate EBS control plane partitioned to the affected zone and finer-grained throttles; the backlog was processed from 11:30 AM PDT on 23 April, and API access in the zone was restored at 6:15 PM PDT. The remaining 2.2% were restored from S3 snapshots taken early in the event, leaving 1.04% by 12:30 PM PDT on 24 April, and the team then ran forensics on the volumes on failed machines that had no snapshot and began restoring them at 3:00 PM PDT; 0.07% could not be restored in a consistent state. For RDS, 2.5% of multi-AZ instances in the Region did not fail over because the fast sequence of the network partition and stuck I/O on the primary hit a previously unseen bug that left the primary isolated, so automatic failover was not safe without risking data loss.

Severity
–
Scope
EBS volumes in one Availability Zone (about 13% stuck, 0.07% not restorable in a consistent state) and the EC2 instances using them; EBS APIs and launches of EBS-backed EC2 instances across the Region; RDS single-AZ instances in the zone (peak 45% stuck) and 2.5% of multi-AZ instances in the Region that did not fail over
Users affected
–
Region
US East Region, one Availability Zone
Time to detect
–
Time to resolve
–

Symptoms

From 12:47 AM PDT on 21 April 2011, a subset of Amazon EBS volumes in one Availability Zone of the US East Region could no longer serve reads or writes, and EC2 instances that tried to use them got stuck as well. AWS called these "stuck" volumes. EBS API calls across the whole US East Region also returned high error rates and latencies during two periods on the first day.

The EBS nodes in the affected cluster had lost both of the networks they use to talk to each other: the high-bandwidth primary network and the lower-capacity replication network, which is meant only as a backup. Many nodes were completely isolated from the rest of the cluster and lost contact with the nodes holding their replicas. When connectivity came back, the nodes all looked for free space to create new replicas at once, and about 13% of the volumes in the Availability Zone ended up stuck. Because data on a volume is blocked while a new primary replica is chosen, an instance doing I/O on such a volume simply hung.

The trouble spread beyond the one zone through the regional EBS control plane. The cluster could not answer "create volume" requests, and because that API had a long time-out, slow calls piled up until the control plane's regional thread pool was full and it failed requests for other Availability Zones too. From 5:30 AM PDT the volume of negotiations with the control plane about which replica is primary, increased by retries and by nodes crashing on a race condition, browned out the control plane again. Launches of new EBS-backed EC2 instances in other zones had elevated error rates for about 11 hours, until noon PDT; AWS's alarming was not fine-grained enough for the one control plane API involved, so those launch errors were hidden behind the general errors from the degraded cluster. Some customers also reported stuck volumes in other zones; AWS measured a peak of less than 0.07% there, caused by slower normal re-mirroring while the control plane was degraded.

Amazon RDS depends on EBS, so RDS databases in the zone were affected too. At the peak 45% of single-AZ database instances in the affected zone had stuck I/O, a larger share than for EBS volumes because each database uses several volumes. Of multi-AZ database instances in the US East Region, 2.5% did not fail over automatically and needed manual intervention. In the end 0.07% of the volumes in the affected zone could not be restored in a consistent state, and 0.4% of single-AZ database instances there had a volume that could not be recovered.

Symptom Storage disconnected or stale mountsNetwork unreachableControl plane lagHigh error rateFailover failureData loss

Root cause

Root cause Operator error

Component Cloud block storageDatabase
Contributing factor Automation amplificationRetry stormNo headroomExcessive blast radiusSignal buried in noise
Trigger
During a routine network change to upgrade the capacity of the primary EBS network in one Availability Zone, traffic was to be shifted off one of the redundant routers. The shift was executed incorrectly and sent the traffic onto the lower-capacity replication network instead of the other primary router, which could not carry it.

Detection

–

Remediation

Action items

Lessons

AWS writes that with more spare capacity the cluster would have absorbed the re-mirroring requests without a storm, and that it now understands how much capacity large recovery events need. The event also showed that the EBS control plane let a failure in one Availability Zone affect the others, so more work is needed to make zones behave as if completely independent. AWS adds that some customers had critical components in a single zone, and that multi-AZ deployments need to become easier to design and run.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud block storage, Database among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.