Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases9 cases

9 to 9 of 9

  1. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster
    Amazon Web Services2014
    Symptoms

    From 9:16 PM to 11:16 PM PDT on June 13, Amazon SimpleDB in the US East Region was unavailable to all API calls except a fraction of eventually consistent reads, and SimpleDB API requests returned server-side errors (http response code 500).

    Service outageHigh error rate
    Trigger

    Power was lost to the servers that hosted several SimpleDB storage nodes in a single data center, and those storage nodes became unavailable at the same time.

    Root cause
    Software defect
    Component
    DatabaseCoordination and key-value storeData center power and cooling
    Contributing factor
    Automation amplification
    Remediation
    • Raised the lock service handshake timeout values by hand
    • Restarted a subset of metadata nodes, which could then authorize the storage nodes to rejoin the cluster
    • Throttled CreateDomain and DeleteDomain calls until 1:30 AM while the remaining metadata nodes recovered
    Action items
    • Set a longer lock service handshake timeout in all SimpleDB Regions
    • Make nodes wait and retry handshakes later with a longer timeout instead of removing themselves from the cluster after repeated timeouts
    Lessons

    The short handshake timeout had been chosen to detect replica failures quickly, and AWS concluded it was too low. Nodes removing themselves from the cluster right after several handshake timeouts widened the event and caused the API errors; the nodes should have waited and retried with a longer timeout.

    View case
Check your cluster for this

These 9 cases involve Coordination and key-value store and Service discovery. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Configuration change2 cases

Component
Coordination and key-value store2Database proxy and connection pooler1Database replication1Pod termination1Service discovery1
Contributing factor
Excessive blast radius1Unknown limit1Retry storm1
Organisation
DigitalOcean1Discord1
  1. IL-0499 Discord voice and video calls hung at Awaiting Endpoint for three hours after 17% of sessions stopped at once
    Discord2026
  2. IL-0374 DigitalOcean creates and management actions failed globally when its service discovery datastore lost quorum
    DigitalOcean2024

Cloud provider outage1 case

Component
Coordination and key-value storeSecrets store
Contributing factor
Automation amplification
Organisation
Python Software Foundation
  1. IL-0514 PyPI's backends stopped responding for 7 minutes as rescheduled pods could not start while Consul and Vault were down
    Python Software Foundation2022

Software defect5 cases

Component
Coordination and key-value store5Service discovery3In-memory cache1conntrack table1Container runtime1Database1Data center power and cooling1Elasticsearch or OpenSearch1Linux kernel1
Contributing factor
Automation amplification2Harmful manual action2Monitoring blind spot2Overlapping failures2Retry storm2Change with a delayed effect1Unpatched known bug1Health check tied to a dependency1Single point of failure1
Organisation
Elastic2Amazon Web Services1Discord1Roblox1
  1. IL-0213 Roblox was down for 73 hours because key-value writes on its single Consul cluster slowed from under 300 ms to 2 seconds
    Roblox2021
  2. IL-0271 An empty etcd service entry crashed Discord's Elixir chat nodes, and Discord was down for most users for an hour
    Discord2020
  3. IL-0277 Patching Elastic Cloud's ZooKeeper ensemble in us-east-1 lost quorum, and hosted clusters and Kibana went unreachable
    Elastic2019
  4. IL-0587 Elastic Cloud proxies in AWS eu-west-1 flooded an overloaded ZooKeeper and ran out of memory, cutting cluster access
    Elastic2019
  5. IL-0536 Amazon SimpleDB in US East returned HTTP 500 to nearly all API calls for two hours after its nodes left the cluster
    Amazon Web Services2014

Faulty deployment1 case

Component
Coordination and key-value store
Contributing factor
Unverified assumptionNo staged rollout
Organisation
Parse.ly
  1. IL-0276 Chef restarted two of three ZooKeeper nodes at once, and Parse.ly's real-time data stopped for about 3 hours
    Parse.ly2016