Incident Library
Applied filters SymptomDatabase overloadFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

8 to 8 of 8

  1. IL-0282 DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service
    Amazon Web Services2015
    Symptoms

    Early on Sunday 20 September 2015, the error rate of customer requests to Amazon DynamoDB in the US-East Region rose far beyond any level of the previous 3 years by 2:37am PDT and settled at about 55%. Some other AWS services in the region that depend on DynamoDB were affected as well.

    High error rateDatabase overload
    Trigger

    At 2:19am PDT a brief network disruption affected a portion of DynamoDB's storage servers, and they all requested their membership data from the metadata service at the same time.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planningMonitoring blind spotRetry stormCircular dependencyExcessive blast radius
    Detection

    For the errors that remained after the main recovery, a growing number of customer support cases on Monday about tables stuck in the updating or deleting stage and higher than normal error rates showed that the low overall error rate was hitting some customers hard.

    Remediation
    • Paused requests to the metadata service, which cut the retries and relieved its load
    • Added significant capacity to the metadata service once it could answer administrative requests
    • Turned requests to the metadata service back on and returned storage servers to the customer request path
    • Restored a metadata partition that was still not taking its full traffic, ending the remaining errors on Monday
    Action items
    • Significantly increased the capacity of the metadata service, which AWS had already done by the time of the summary
    • Add stricter monitoring on performance dimensions such as membership size, to plan capacity ahead
    • Reduce the rate at which storage nodes request membership data
    • Lengthen the time allowed to process membership queries
    • Segment DynamoDB into many metadata service instances, each serving only part of the storage server fleet
    • Change SQS metadata caching so send and receive keep working without long access to the metadata table
    • Partition work across the Auto Scaling servers to process a backlog of scaling jobs in parallel
    • Prune older scaling activities that have been superseded
    • Increase the capacity available to process scaling activities
    • Change CloudWatch caching of DynamoDB group membership data to refresh only the smallest set of metrics
    • Deliver CloudWatch metrics faster through write-through caching
    • Roll out the shorter timeout on the DynamoDB-dependent API call in Console login
    Lessons

    A low overall error rate can hide disproportionately high error rates for a few customers, and AWS should have posted to the dashboard sooner. A call that does not need to succeed, like the DynamoDB-dependent API call in Console login, should fail quickly instead of blocking on a very long timeout. Splitting the metadata service into many instances, each serving part of the fleet, is meant to contain the impact of software, capacity or infrastructure failures.

    View case
Check your cluster for this

These 8 cases involve Database and PostgreSQL vacuum. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause4

Breaking change in an upgrade3 cases

Component
Database3Database proxy and connection pooler2PostgreSQL vacuum1Database replication1Message broker1Query planner and table statistics1Serverless function platform1
Contributing factor
Monitoring blind spot3Unverified assumption1Automation amplification1Unreviewed defaults1Change with a delayed effect1Unknown limit1Harmful manual action1Overlapping failures1Retry storm1Unrepresentative staging1Unpinned version1
Organisation
Clerk1Mercari1RevenueCat1
  1. IL-0572 About 10 million Pub/Sub messages backed up and Merpay accounting data was not written to Cloud Spanner for over a day
    Mercari2026
  2. IL-0498 Clerk's Postgres database had load spikes every 15 minutes and requests failed and slowed intermittently for four days
    Clerk2025
  3. IL-0308 Most RevenueCat backend systems failed after cutover to a new Aurora PostgreSQL 14 cluster as queries became very slow
    RevenueCat2022

Query plan regression1 case

Component
DatabaseQuery planner and table statistics
Contributing factor
Inefficient queryMonitoring blind spot
Organisation
Clerk
  1. IL-0317 Clerk's Postgres slowed after an automatic analyze flipped a query plan, and over 95% of requests got 429
    Clerk2026

Faulty deployment1 case

Component
Database
Contributing factor
Unverified assumptionChange with a delayed effectInefficient queryMonitoring blind spotUnrepresentative staging
Organisation
STORES
  1. IL-0571 STORES reservation returned bursts of 504 errors when a smart-list query held a database process for over 20 minutes
    STORES2024

Capacity exhaustion3 cases

Component
Database3PostgreSQL vacuum1
Contributing factor
Monitoring blind spot3Excessive blast radius2No capacity planning2Circular dependency1Inefficient query1Harmful manual action1Retry storm1
Organisation
37signals1Amazon Web Services1Inflab1
  1. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice
    37signals2022
  2. IL-0456 Inflearn went down six times during a January 2022 free-course event as slow queries piled up on Aurora PostgreSQL
    Inflab2022
  3. IL-0282 DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service
    Amazon Web Services2015