Incident Library
Applied filters ComponentCloud load balancerSymptomHigh latencyFactorUntested backupClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0224 ELB control plane APIs failed in US-East, and load balancers it then changed broke, after ELB state data was deleted
    Amazon Web Services2012
    Symptoms

    From 12:24 PM PST on 24 December 2012, the Amazon Elastic Load Balancing (ELB) control plane in the US-East Region showed high latency and error rates for API calls that manage load balancers. Running load balancers kept handling requests normally at first.

    Control plane lagHigh error rateHigh latency
    Trigger

    One of the very few developers with access to the production ELB environment inadvertently ran a maintenance process against the production ELB state data, and it logically deleted part of that data.

    Root cause
    Operator error
    Component
    Cloud load balancer
    Contributing factor
    No safeguard on destructive changeUntested backupAutomation amplification
    Detection

    The ELB team noticed the control plane API errors first; the missing state data was found only after the team investigated load balancers that had become degraded.

    Remediation
    • Disabled several ELB control plane workflows, including scaling and descaling
    • Recovered some of the affected running load balancers by hand
    • Restored a snapshot of the ELB state data from just before the deletion, using an alternate recovery process
    • Merged the restored data with the state changes since the snapshot and verified it
    • Re-enabled the ELB workflows and APIs slowly while recovering each affected load balancer
    Action items
    • Require per-incident Change Management approval for all access to production ELB state data
    • Change the data recovery process so ELB state data can be recovered much faster
    • Reprogram the ELB control plane workflows to reconcile central service data with current load balancer state
    View case
Check your cluster for this

This case involves Cloud load balancer. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Operator error1 case

Component
Cloud load balancer
Contributing factor
Automation amplificationUntested backupNo safeguard on destructive change
Organisation
Amazon Web Services
  1. IL-0224 ELB control plane APIs failed in US-East, and load balancers it then changed broke, after ELB state data was deleted
    Amazon Web Services2012