Incident Library
Applied filters Root causeCapacity exhaustionSymptomDatabase overloadFactorHarmful manual actionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice
    37signals2022
    Symptoms

    At 13:00 UTC on Monday, June 27, 2022, Basecamp 3 became extremely slow, then showed error pages, and then stopped working altogether. Basecamp 2 and Basecamp Classic kept working normally.

    Lock contention or deadlockDatabase overloadHigh latencyHigh error rateService outage
    Trigger

    At 13:00 UTC, when Basecamp 3 sends scheduled notifications in massive batches, a rush of demanding queries reached the primary database and started a cascade of heavy lock contention.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No capacity planningHarmful manual actionMonitoring blind spot
    Detection

    The team saw the Basecamp 3 primary database become completely congested within 30 seconds of 13:00 UTC and responded immediately.

    Remediation
    • Paused and disabled parts of Basecamp 3, such as delaying progress report emails
    • Failed over from the primary database to a replica, which brought Basecamp 3 back until it tipped over again
    • Put Basecamp 3 into maintenance mode
    • Set hard limits on database query time
    • Brought features back online step by step with the query limits in place
    Action items
    • Keep hard limits on database query time so slow queries fail instead of piling up
    • Tune the limits on concurrency
    • Treat top-of-the-hour database load spikes from scheduled batches as unacceptable
    • Reduce the number of parallel workers for some scheduled jobs
    • Let some scheduled activities run over several minutes instead of all at once
    • Move bursty queries from the primary database to read-only replicas
    • Fix the quiet performance regressions found in a review of the past year of database metrics
    • Find and spread out other bursty activities that concentrate load on the database
    • Tune the database for more predictable behaviour under heavy load and high concurrency
    • Add observability into the database's internal locking and transaction states
    • Build tools to switch off high-load Basecamp 3 features selectively during an incident
    • Run operational fire drills for a wedged database
    • Evaluate upgrading the database software and hardware
    • Set service level objectives for the components behind Basecamp 3
    • Hold quarterly performance and load reviews to catch slow, subtle changes
    Lessons

    A system running just past its limits can flip from stable to unstable without any particular trigger, and steady growth plus small regressions that raised no alarm had brought Basecamp 3's primary database to that point. Under lock contention, terminating queries does not relieve the database, because rolling back their transactions waits on the same locks. The team struggled to respond with confidence because there was no single clear cause and little visibility into the database's internals.

    View case
  2. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022
    Symptoms

    On March 16, 2022 at 14:09 UTC, write operations across GitHub stopped working, including git operations, webhooks, pull requests, API requests, issues, GitHub Packages, GitHub Codespaces, GitHub Actions and GitHub Pages. Under peak-hour load, the database proxy in front of the shared mysql1 MySQL database had reached its maximum number of connections.

    Writes rejectedDatabase overloadHigh error rateFailover failure
    Trigger

    Increased load on mysql1 during peak hours, combined with poor query performance in specific circumstances, pushed the database proxy to its maximum number of connections.

    Root cause
    Capacity exhaustion
    Component
    DatabaseDatabase proxy and connection pooler
    Contributing factor
    Inefficient queryChange during peak hoursHarmful manual actionExcessive blast radius
    Remediation
    • Failed over mysql1 to a healthy replica on March 16
    • Reset the application connections to the new primary on March 17
    • Added an index that fixed the main query performance problem
    • Performed a primary failover on March 22
    • Performed a primary failover on March 23
    • Throttled webhook traffic to reduce load
    Action items
    • Keep throttling webhook traffic at peak load times
    • Audit the load patterns of mysql1 during peak hours and make performance fixes based on the audits
    • Move traffic to other databases to reduce load and speed up failover
    • Review change management, particularly monitoring and changes during high load in production
    • Shard the databases and scale up hardware
    View case
  3. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
    PagerDuty2014sev1
    Symptoms

    On 3 June 2014, PagerDuty's notification pipeline performed poorly and some notifications were delayed after the Cassandra cluster behind it became heavily degraded. About 3% of events sent to the integration API could not be received.

    Database overloadHigh error rateHigh latency
    Trigger

    On 3 June 2014 the Cassandra background repair process started its normal run on one node of the cluster while other high load was on the cluster; on 4 June the team restarted the postponed repair by hand.

    Root cause
    Capacity exhaustion
    Component
    Database
    Contributing factor
    No headroomExcessive blast radiusHarmful manual actionIgnored early signal
    Remediation
    • Reduced the load on the Cassandra cluster
    • Stopped the Cassandra repair process
    • Cut communication between some Cassandra nodes to stabilise the cluster
    • On 4 June, deleted all in-flight data in the notification pipeline in a "factory reset"
    • Brought downstream systems back into a consistent state with the emptied pipeline by hand
    Action items
    • Scale the existing Cassandra nodes vertically to bigger, faster servers
    • Set up multiple Cassandra clusters and spread the workloads across them
    • Set system load thresholds at which the Cassandra clusters are scaled out horizontally ahead of need
    • Upgrade the current and new clusters to a more recent Cassandra version
    • Add more load shedding techniques to control Cassandra under high load
    • Bring more Cassandra expertise in-house
    Lessons

    PagerDuty concluded that its Cassandra cluster was underscaled and shared by too many services with different load and usage patterns, and that the scaling work should have come before the planned Cassandra version upgrade.

    View case
Check your cluster for this

These 3 cases involve Database and Database proxy and connection pooler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion3 cases

Component
Database3Database proxy and connection pooler1
Contributing factor
Harmful manual action3Excessive blast radius2Change during peak hours1Ignored early signal1Inefficient query1Monitoring blind spot1No capacity planning1No headroom1
Organisation
37signals1GitHub1PagerDuty1
  1. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice
    37signals2022
  2. IL-0302 Peak load on GitHub's mysql1 MySQL cluster maxed out its database proxy's connections and stopped writes in March 2022
    GitHub2022
  3. IL-0383 PagerDuty's shared Cassandra cluster degraded during a repair, delaying notifications and losing incoming events
    PagerDuty2014