Incident Library

IL-0224 Amazon Web Services · 2012-12-24awsnetworking

ELB control plane APIs failed in US-East, and load balancers it then changed broke, after ELB state data was deleted

The ELB control plane uses state data to manage the configuration of the load balancers in a region, for example which backend hosts each load balancer sends traffic to. A maintenance process run against production deleted part of this data at 12:24 PM PST on 24 December, and the developer who ran it did not notice. AWS normally blocks all access to production service data, but the ELB service had given a few developers extra access for operational processes that were being automated, and that access had been set as persistent instead of requiring approval for each use. The missing data was not needed for basic operation, so running load balancers kept working until the control plane changed them; then the control plane, lacking the data, configured them wrongly.

At 5:02 PM PST the team disabled several ELB control plane workflows, including scaling and descaling, so that no more running load balancers would be changed; the rest kept working but could not scale or be modified by customers. Some affected load balancers were recovered by hand that night. The first method used to restore the state data to just before 12:24 PM took several hours and did not produce a usable snapshot. With another process, a snapshot from just before the deletion was restored at 2:45 AM PST on 25 December, merged with the state changes since then and verified by 5:40 AM. Workflows and APIs were then re-enabled slowly: most by 8:15 AM, almost all affected load balancers fully working by 10:30 AM, and normal operation announced at 12:05 PM PST.

Severity
–
Scope
Applications using Amazon ELB in the US-East Region: up to 6.8% of running load balancers degraded with errors, and the rest could not scale or be modified
Users affected
–
Region
US-East
Time to detect
–
Time to resolve
22h 6m

Symptoms

From 12:24 PM PST on 24 December 2012, the Amazon Elastic Load Balancing (ELB) control plane in the US-East Region showed high latency and error rates for API calls that manage load balancers. Running load balancers kept handling requests normally at first.

Some APIs succeeded and others failed: customers could create and manage new load balancers but could not manage existing ones. This puzzled the ELB team, which spent the next couple of hours on the API errors.

Later some running load balancers began to perform badly and return errors for the customer applications behind them. This happened only after the ELB control plane had changed a load balancer, because a user modified its configuration or because it had to scale up or down; at the peak, 6.8% of running ELB load balancers were affected, and the rest operated correctly but could not scale or be modified by customers. Only when the team dug into these degraded load balancers did it find that part of the ELB state data was missing.

Symptom Control plane lagHigh error rateHigh latency

Root cause

Root cause Operator error

Component Cloud load balancer
Contributing factor No safeguard on destructive changeUntested backupAutomation amplification
Trigger
One of the very few developers with access to the production ELB environment inadvertently ran a maintenance process against the production ELB state data, and it logically deleted part of that data.

Detection

The ELB team noticed the control plane API errors first; the missing state data was found only after the team investigated load balancers that had become degraded.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Cloud load balancer among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.