Cases1 case

1 to 1 of 1

1. IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover

   37signals2020

   Symptoms

   On Tuesday 1 September 2020 Basecamp could not be loaded at all for 17 minutes, from 9:51am to 10:08am Central Time (14:51–15:08 UTC), and nothing seemed to work. On Wednesday 2 September it happened again for 15 minutes, from 10:50am to 11:05am Central Time (15:50–16:05 UTC).

   Service outageLink flappingHigh latency

   Trigger

   On 1 September the network links between the data centers went offline, and during the recovery a hardware load balancer crashed. On 2 September the primary load balancer in Virginia crashed again, and later its secondary crashed too.

   Root cause

   Software defect

   Component

   On-premises load balancerBackbone network (WAN)

   Contributing factor

   Overlapping failuresNo headroom

   Remediation

   • On 1 September, intervened by hand to get the network routers working with the standby load balancer
   • On 2 September, fixed the same router and load balancer desynchronization by hand

   Action items

   • Find the cause of the load balancer crash with the vendor and apply its bugfix
   • Replace the hardware load balancers
   • Reroute the redundant cross-datacenter network paths for proper circuit diversity
   • Evaluate moving from hardware to software load balancers, which can be deployed in minutes
   • Evaluate decentralizing the load balancer architecture to limit the impact of one failure
   • Assess the systems for resilience, fragility and risk, and review the assessment process itself

   Lessons

   Basecamp plans for failures like these and practises responses to catastrophic failures in its live systems, but three obscure issues in five days chained into overlapping failures. What caught the team by surprise were cascading failures that exposed unexpected fragility and hard paths to recovery, which Basecamp says it can prepare for.

   View case

By root cause1