Cases3 cases

1 to 3 of 3

1. IL-0301 Basecamp 3's primary database went into total gridlock at the top of the hour and Basecamp 3 went down twice

   37signals2022

   Symptoms

   At 13:00 UTC on Monday, June 27, 2022, Basecamp 3 became extremely slow, then showed error pages, and then stopped working altogether. Basecamp 2 and Basecamp Classic kept working normally.

   Lock contention or deadlockDatabase overloadHigh latencyHigh error rateService outage

   Trigger

   At 13:00 UTC, when Basecamp 3 sends scheduled notifications in massive batches, a rush of demanding queries reached the primary database and started a cascade of heavy lock contention.

   Root cause

   Capacity exhaustion

   Component

   Database

   Contributing factor

   No capacity planningHarmful manual actionMonitoring blind spot

   Detection

   The team saw the Basecamp 3 primary database become completely congested within 30 seconds of 13:00 UTC and responded immediately.

   Remediation

   • Paused and disabled parts of Basecamp 3, such as delaying progress report emails
   • Failed over from the primary database to a replica, which brought Basecamp 3 back until it tipped over again
   • Put Basecamp 3 into maintenance mode
   • Set hard limits on database query time
   • Brought features back online step by step with the query limits in place

   Action items

   • Keep hard limits on database query time so slow queries fail instead of piling up
   • Tune the limits on concurrency
   • Treat top-of-the-hour database load spikes from scheduled batches as unacceptable
   • Reduce the number of parallel workers for some scheduled jobs
   • Let some scheduled activities run over several minutes instead of all at once
   • Move bursty queries from the primary database to read-only replicas
   • Fix the quiet performance regressions found in a review of the past year of database metrics
   • Find and spread out other bursty activities that concentrate load on the database
   • Tune the database for more predictable behaviour under heavy load and high concurrency
   • Add observability into the database's internal locking and transaction states
   • Build tools to switch off high-load Basecamp 3 features selectively during an incident
   • Run operational fire drills for a wedged database
   • Evaluate upgrading the database software and hardware
   • Set service level objectives for the components behind Basecamp 3
   • Hold quarterly performance and load reviews to catch slow, subtle changes

   Lessons

   A system running just past its limits can flip from stable to unstable without any particular trigger, and steady growth plus small regressions that raised no alarm had brought Basecamp 3's primary database to that point. Under lock contention, terminating queries does not relieve the database, because rolling back their transactions waits on the same locks. The team struggled to respond with confidence because there was no single clear cause and little visibility into the database's internals.

   View case
2. IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover

   37signals2020

   Symptoms

   On Tuesday 1 September 2020 Basecamp could not be loaded at all for 17 minutes, from 9:51am to 10:08am Central Time (14:51–15:08 UTC), and nothing seemed to work. On Wednesday 2 September it happened again for 15 minutes, from 10:50am to 11:05am Central Time (15:50–16:05 UTC).

   Service outageLink flappingHigh latency

   Trigger

   On 1 September the network links between the data centers went offline, and during the recovery a hardware load balancer crashed. On 2 September the primary load balancer in Virginia crashed again, and later its secondary crashed too.

   Root cause

   Software defect

   Component

   On-premises load balancerBackbone network (WAN)

   Contributing factor

   Overlapping failuresNo headroom

   Remediation

   • On 1 September, intervened by hand to get the network routers working with the standby load balancer
   • On 2 September, fixed the same router and load balancer desynchronization by hand

   Action items

   • Find the cause of the load balancer crash with the vendor and apply its bugfix
   • Replace the hardware load balancers
   • Reroute the redundant cross-datacenter network paths for proper circuit diversity
   • Evaluate moving from hardware to software load balancers, which can be deployed in minutes
   • Evaluate decentralizing the load balancer architecture to limit the impact of one failure
   • Assess the systems for resilience, fragility and risk, and review the assessment process itself

   Lessons

   Basecamp plans for failures like these and practises responses to catastrophic failures in its live systems, but three obscure issues in five days chained into overlapping failures. What caught the team by surprise were cascading failures that exposed unexpected fragility and hard paths to recovery, which Basecamp says it can prepare for.

   View case
3. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat

   37signals2020

   Symptoms

   At 11:59AM CDT on Friday 28 August 2020, Basecamp's Ops team received a flood of alerts from PagerDuty, Nagios and Prometheus. Monitoring showed that the newer of the two metro links between the data centers in Ashburn, VA and Chicago, IL had been lost for about 20-30 seconds, and BC3 kept responding slowly after the link came back.

   Link flappingHigh latencyReplication lag

   Trigger

   Both redundant metro links between Basecamp's Ashburn and Chicago data centers dropped for short periods; monitoring showed the newer link down for about 20-30 seconds.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingIn-memory cacheDatabase replication

   Contributing factor

   Single point of failureRetry stormSignal buried in noise

   Detection

   A flood of alerts from PagerDuty, Nagios and Prometheus, with monitoring showing the lost metro link. A lingering alert for one Redis DB instance later pointed to the replication problem.

   Remediation

   • Turned chat off at the load balancer layer so the rest of BC3 could stabilise
   • Restarted Redis on the replica that was stuck in the resynchronization loop

   Action items

   • Make the documented Redis configuration changes for the known replication issue
   • Make concrete plans to remove the shared physical path under the two metro links
   • Prove the failover systems in production at peak load
   • Make explicit room for post-incident follow-up work and push other work out

   Lessons

   Planned redundancy and live testing had given Basecamp a false sense of security, especially at peak load: moving between redundant network links or shifting load across data centers should work without issue, and Basecamp would rather find out in a controlled test with the full team ready. The follow-up from an outage needs urgency and explicitly cleared time, not just a place on a list.

   View case

By root cause3