IL-0256 37signals · 2020-09-01networking

Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover

Basecamp runs two redundant data centers connected by two network links, and hardware load balancers direct Internet traffic to its servers. On 1 September 2020 the links between the data centers went offline, as they had on 28 August, and took Basecamp 3 Campfire chats and Pings down. During the recovery one load balancer crashed. The standby load balancer picked up operations at once, but the network routers did not synchronize automatically with it, and Basecamp was unreachable for 17 minutes until the team intervened by hand.

On 2 September the primary load balancer in the Virginia data center crashed in the morning and failed over cleanly. Later the secondary load balancer crashed as well and failed back to the former primary, which caused the same desynchronization between routers and load balancer and another 15 minutes of outage that needed manual intervention, followed by hours of slow chat. Basecamp worked with its load balancer vendor on the crash and had a preliminary assessment and bugfix. Basecamp said it had been pushing the hardware load balancers hard and that traffic overload was a driving factor in one of the outages.

Severity: –
Scope: All of Basecamp could not be loaded for 17 minutes on 1 September and 15 minutes on 2 September; Campfire chats and Pings were down on 1 September and slow for hours after 2 September
Users affected: –
Region: –
Time to detect: –
Time to resolve: –

Symptoms

On Tuesday 1 September 2020 Basecamp could not be loaded at all for 17 minutes, from 9:51am to 10:08am Central Time (14:51–15:08 UTC), and nothing seemed to work. On Wednesday 2 September it happened again for 15 minutes, from 10:50am to 11:05am Central Time (15:50–16:05 UTC).

On 1 September the network links between Basecamp's two data centers went offline first and took Basecamp 3 Campfire chats and Pings down. While the team was recovering from that, one of the hardware load balancers crashed. A standby load balancer took over immediately, but the network routers failed to synchronize automatically with the new load balancer, and the outage lasted until someone intervened by hand.

On 2 September the primary load balancer in the Virginia data center crashed early in the morning and failed over to its secondary as expected. Later that morning the secondary also crashed and failed back to the former primary, and the routers lost synchronization in the same way, again needing manual intervention. After Basecamp came back, chat messages were slow and sluggish for hours.

Symptom Service outage Link flapping High latency

Root cause

Root cause Software defect

Component On-premises load balancer Backbone network (WAN)

Contributing factor Overlapping failures No headroom

Trigger: On 1 September the network links between the data centers went offline, and during the recovery a hardware load balancer crashed. On 2 September the primary load balancer in Virginia crashed again, and later its secondary crashed too.

Detection

–

Remediation

On 1 September, intervened by hand to get the network routers working with the standby load balancer
On 2 September, fixed the same router and load balancer desynchronization by hand

Action items

Find the cause of the load balancer crash with the vendor and apply its bugfix
Replace the hardware load balancers
Reroute the redundant cross-datacenter network paths for proper circuit diversity
Evaluate moving from hardware to software load balancers, which can be deployed in minutes
Evaluate decentralizing the load balancer architecture to limit the impact of one failure
Assess the systems for resilience, fragility and risk, and review the assessment process itself

Lessons

Basecamp plans for failures like these and practises responses to catastrophic failures in its live systems, but three obscure issues in five days chained into overlapping failures. What caught the team by surprise were cascading failures that exposed unexpected fragility and hard paths to recovery, which Basecamp says it can prepare for.

Sources

https://signalvnoise.com/svn3/three-basecamp-outages-one-week-what-happened/ · blog · archived copy

Related cases

IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failedLinode · 2023
same root cause · shared Backbone network (WAN)
IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud · 2019
same root cause · shared Backbone network (WAN)

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, On-premises load balancer, Backbone network (WAN) among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0256
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0256/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.