Incident Library
Applied filters Root causeCapacity exhaustionComponentDatabaseFactorNo capacity planningClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases4 cases

1 to 4 of 4

  1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed
    Buttondown2026
    Symptoms

    On 31 March 2026 Buttondown was down twice, for about seven and about thirteen minutes. New requests could not get a database connection and failed, although the database itself was healthy: queries were fast, CPU and memory were fine and replication lag was nominal.

    Connection pool exhaustionService outage
    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planningMonitoring blind spotCircular dependency
    Detection

    User reports and the team's own manual observation, not automated monitoring. The health check endpoint kept returning 200 for most requests, so no alert fired.

    Remediation
    • Killed active queries to free database connections in the first outage
    • Restarted the database to force-kill the running queries in the second outage
    Action items
    • Reserve a small number of database connections for administrative access
    • Monitor and alert on the database connection count as a percentage of the configured maximum
    • Make the health check endpoint run a lightweight query so connection exhaustion shows in the health checker and the load balancer
    • Add database connection limits to the quarterly capacity review
    • Document and script a fallback that force-recycles workers to release connections without needing a database connection
    Lessons

    Buttondown notes that the tool used to recover from the first outage needed the same resource that was exhausted, which locked the team out during the second outage, and that a health check which cannot detect connection exhaustion, the most common failure mode, gives little protection. A configuration value set long ago drifted out of line with traffic because nobody reviewed it.

    View case
  2. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice
    37signals2022
    Symptoms

    At 13:00 UTC on Monday, June 27, 2022, Basecamp 3 became extremely slow, then showed error pages, and then stopped working altogether. Basecamp 2 and Basecamp Classic kept working normally.

    Lock contention or deadlockDatabase overloadHigh latencyHigh error rateService outage
    Trigger

    At 13:00 UTC, when Basecamp 3 sends scheduled notifications in massive batches, a rush of demanding queries reached the primary database and started a cascade of heavy lock contention.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planningHarmful manual actionMonitoring blind spot
    Detection

    The team saw the Basecamp 3 primary database become completely congested within 30 seconds of 13:00 UTC and responded immediately.

    Remediation
    • Paused and disabled parts of Basecamp 3, such as delaying progress report emails
    • Failed over from the primary database to a replica, which brought Basecamp 3 back until it tipped over again
    • Put Basecamp 3 into maintenance mode
    • Set hard limits on database query time
    • Brought features back online step by step with the query limits in place
    Action items
    • Keep hard limits on database query time so slow queries fail instead of piling up
    • Tune the limits on concurrency
    • Treat top-of-the-hour database load spikes from scheduled batches as unacceptable
    • Reduce the number of parallel workers for some scheduled jobs
    • Let some scheduled activities run over several minutes instead of all at once
    • Move bursty queries from the primary database to read-only replicas
    • Fix the quiet performance regressions found in a review of the past year of database metrics
    • Find and spread out other bursty activities that concentrate load on the database
    • Tune the database for more predictable behaviour under heavy load and high concurrency
    • Add observability into the database's internal locking and transaction states
    • Build tools to switch off high-load Basecamp 3 features selectively during an incident
    • Run operational fire drills for a wedged database
    • Evaluate upgrading the database software and hardware
    • Set service level objectives for the components behind Basecamp 3
    • Hold quarterly performance and load reviews to catch slow, subtle changes
    Lessons

    A system running just past its limits can flip from stable to unstable without any particular trigger, and steady growth plus small regressions that raised no alarm had brought Basecamp 3's primary database to that point. Under lock contention, terminating queries does not relieve the database, because rolling back their transactions waits on the same locks. The team struggled to respond with confidence because there was no single clear cause and little visibility into the database's internals.

    View case
  3. IL-0300 Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,647
    37signals2018
    Symptoms

    On November 9th, 2018, from 7:21am to 12:11pm CST, Basecamp 3 was in read-only mode. Users could open existing messages, to-do lists and files, but could not add anything new or change anything that was already there.

    Writes rejected
    Trigger

    The busy events table, which records almost every activity in Basecamp 3, reached its ceiling of 2,147,483,647, and new events could not be written.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planning
    Remediation
    • Raised the ceiling of the tables from 2,147,483,647 to 9,223,372,036,854,775,807, the same change as the Rails 5.1 default
    Lessons

    The outage was avoidable. The team did not account for how quickly the events table was running out of headroom while it was already working on the problem, and it should have reviewed its own database schema when Rails 5.1 announced the new default in 2017.

    View case
  4. IL-0282 DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service
    Amazon Web Services2015
    Symptoms

    Early on Sunday 20 September 2015, the error rate of customer requests to Amazon DynamoDB in the US-East Region rose far beyond any level of the previous 3 years by 2:37am PDT and settled at about 55%. Some other AWS services in the region that depend on DynamoDB were affected as well.

    High error rateDatabase overload
    Trigger

    At 2:19am PDT a brief network disruption affected a portion of DynamoDB's storage servers, and they all requested their membership data from the metadata service at the same time.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planningMonitoring blind spotRetry stormCircular dependencyExcessive blast radius
    Detection

    For the errors that remained after the main recovery, a growing number of customer support cases on Monday about tables stuck in the updating or deleting stage and higher than normal error rates showed that the low overall error rate was hitting some customers hard.

    Remediation
    • Paused requests to the metadata service, which cut the retries and relieved its load
    • Added significant capacity to the metadata service once it could answer administrative requests
    • Turned requests to the metadata service back on and returned storage servers to the customer request path
    • Restored a metadata partition that was still not taking its full traffic, ending the remaining errors on Monday
    Action items
    • Significantly increased the capacity of the metadata service, which AWS had already done by the time of the summary
    • Add stricter monitoring on performance dimensions such as membership size, to plan capacity ahead
    • Reduce the rate at which storage nodes request membership data
    • Lengthen the time allowed to process membership queries
    • Segment DynamoDB into many metadata service instances, each serving only part of the storage server fleet
    • Change SQS metadata caching so send and receive keep working without long access to the metadata table
    • Partition work across the Auto Scaling servers to process a backlog of scaling jobs in parallel
    • Prune older scaling activities that have been superseded
    • Increase the capacity available to process scaling activities
    • Change CloudWatch caching of DynamoDB group membership data to refresh only the smallest set of metrics
    • Deliver CloudWatch metrics faster through write-through caching
    • Roll out the shorter timeout on the DynamoDB-dependent API call in Console login
    Lessons

    A low overall error rate can hide disproportionately high error rates for a few customers, and AWS should have posted to the dashboard sooner. A call that does not need to succeed, like the DynamoDB-dependent API call in Console login, should fail quickly instead of blocking on a very long timeout. Splitting the metadata service into many instances, each serving part of the fleet, is meant to contain the impact of software, capacity or infrastructure failures.

    View case
Check your cluster for this

These 4 cases involve Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion4 cases

Component
Database4
Contributing factor
No capacity planning4Monitoring blind spot3Circular dependency2Excessive blast radius1Harmful manual action1Retry storm1
Organisation
37signals2Amazon Web Services1Buttondown1
  1. IL-0497 Buttondown's database ran out of connections twice in one day and requests failed while health checks mostly passed
    Buttondown2026
  2. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice
    37signals2022
  3. IL-0300 Basecamp 3 went read-only for almost five hours when its events table reached the ceiling of 2,147,483,647
    37signals2018
  4. IL-0282 DynamoDB in US-East reached about 55% request errors as storage servers overloaded its metadata service
    Amazon Web Services2015