Incident Library
Applied filters SymptomService outageFactorNo capacity planningOrganisation37signalsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice
    37signals2022
    Symptoms

    At 13:00 UTC on Monday, June 27, 2022, Basecamp 3 became extremely slow, then showed error pages, and then stopped working altogether. Basecamp 2 and Basecamp Classic kept working normally.

    Lock contention or deadlockDatabase overloadHigh latencyHigh error rateService outage
    Trigger

    At 13:00 UTC, when Basecamp 3 sends scheduled notifications in massive batches, a rush of demanding queries reached the primary database and started a cascade of heavy lock contention.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planningHarmful manual actionMonitoring blind spot
    Detection

    The team saw the Basecamp 3 primary database become completely congested within 30 seconds of 13:00 UTC and responded immediately.

    Remediation
    • Paused and disabled parts of Basecamp 3, such as delaying progress report emails
    • Failed over from the primary database to a replica, which brought Basecamp 3 back until it tipped over again
    • Put Basecamp 3 into maintenance mode
    • Set hard limits on database query time
    • Brought features back online step by step with the query limits in place
    Action items
    • Keep hard limits on database query time so slow queries fail instead of piling up
    • Tune the limits on concurrency
    • Treat top-of-the-hour database load spikes from scheduled batches as unacceptable
    • Reduce the number of parallel workers for some scheduled jobs
    • Let some scheduled activities run over several minutes instead of all at once
    • Move bursty queries from the primary database to read-only replicas
    • Fix the quiet performance regressions found in a review of the past year of database metrics
    • Find and spread out other bursty activities that concentrate load on the database
    • Tune the database for more predictable behaviour under heavy load and high concurrency
    • Add observability into the database's internal locking and transaction states
    • Build tools to switch off high-load Basecamp 3 features selectively during an incident
    • Run operational fire drills for a wedged database
    • Evaluate upgrading the database software and hardware
    • Set service level objectives for the components behind Basecamp 3
    • Hold quarterly performance and load reviews to catch slow, subtle changes
    Lessons

    A system running just past its limits can flip from stable to unstable without any particular trigger, and steady growth plus small regressions that raised no alarm had brought Basecamp 3's primary database to that point. Under lock contention, terminating queries does not relieve the database, because rolling back their transactions waits on the same locks. The team struggled to respond with confidence because there was no single clear cause and little visibility into the database's internals.

    View case
Check your cluster for this

This case involves Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion1 case

Component
Database
Contributing factor
Harmful manual actionMonitoring blind spotNo capacity planning
Organisation
37signals
  1. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice
    37signals2022