IL-0255 37signals · 2020-08-28networkingdatabase
Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
Basecamp has redundant metro links between its primary data center in Ashburn, VA and its other data center in Chicago, IL. A few months before the incident it changed networking providers for these links after its previous vendor was acquired, ran the new provider alongside the old one for several weeks, and switched over entirely in late June 2020.
On 28 August the short loss of connectivity between the data centers led to replication errors on a Redis standby server, and the replica fell into a resynchronization loop that kept taking resources from the primary and made it very slow. Chat reconnections added load that slowed the whole of BC3. The team turned chat off at the load balancer layer to stabilise the rest of BC3, and restarting Redis on the replica brought an immediate improvement, after which BC3 returned to normal. The Redis behaviour was a known issue with a documented workaround that was new to the team; the Ops lead created a todo for the configuration change but did not make sure it was assigned or finished before the next incident.
The question afterwards was how a 30 second loss of one redundant link could take BC3 down. Logs showed that both links had dropped. The providers first pointed to sub-optimal BGP configuration settings, which did not fully explain the loss of both circuits. When the links dropped again on 1 September, the providers watched it happen and found that both metro links share a physical path in Pennsylvania that was affected by a fiber cut, which surprised Basecamp and the providers alike.
- Severity
- –
- Scope
- BC3: slow responses throughout, and chats and pings not working
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- –
Symptoms
At 11:59AM CDT on Friday 28 August 2020, Basecamp's Ops team received a flood of alerts from PagerDuty, Nagios and Prometheus. Monitoring showed that the newer of the two metro links between the data centers in Ashburn, VA and Chicago, IL had been lost for about 20-30 seconds, and BC3 kept responding slowly after the link came back.
Chats and pings in BC3 then stopped working entirely, and chat reconnections were overloading the network and slowing all of BC3. Because the problem was clearly tied to chat, the team restarted the Cable service, which did not fix the connections. Turning chat off at the load balancer layer let the rest of BC3 settle, but a second Cable restart had no effect.
When the alert noise died down, one alert for a single Redis DB instance remained. The team had passed over it at first because the database was not down, and probes from the command line still got answers. Further digging found replication errors on a standby server: its replica was stuck in a resynchronization loop that kept taking resources and slowed the primary so much that it answered little more than monitoring checks.
Logs reviewed after the incident showed that both metro links, not only the newer one, had dropped for short periods.
Root cause
Root cause Dependency failure
- Trigger
- Both redundant metro links between Basecamp's Ashburn and Chicago data centers dropped for short periods; monitoring showed the newer link down for about 20-30 seconds.
Detection
A flood of alerts from PagerDuty, Nagios and Prometheus, with monitoring showing the lost metro link. A lingering alert for one Redis DB instance later pointed to the replication problem.
Remediation
- Turned chat off at the load balancer layer so the rest of BC3 could stabilise
- Restarted Redis on the replica that was stuck in the resynchronization loop
Action items
- Make the documented Redis configuration changes for the known replication issue
- Make concrete plans to remove the shared physical path under the two metro links
- Prove the failover systems in production at peak load
- Make explicit room for post-incident follow-up work and push other work out
Lessons
Planned redundancy and live testing had given Basecamp a false sense of security, especially at peak load: moving between redundant network links or shifting load across data centers should work without issue, and Basecamp would rather find out in a controlled test with the full team ready. The follow-up from an outage needs urgency and explicitly cleared time, not just a place on a list.
Sources
Related cases
IL-0267Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were downWikimedia Foundation · 2023same root cause · shared BGP routing, Backbone network (WAN)
IL-0358Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed trafficPacket · 2019same root cause · shared BGP routing, Backbone network (WAN)
IL-0408OpenRouter's API returned 500s and false 401 "User not found" errors when its API key cache dropped and came back emptyOpenRouter · 2026same root cause · shared In-memory cache
IL-0526OpenStreetMap.org went offline, then read-only, for 56 hours after HE.net routing equipment cut both fibre uplinksOpenStreetMap Foundation · 2024same root cause · shared Database replication
IL-0509SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnetSourceHut · 2024same root cause · shared BGP routing
IL-0258FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went downFastmail · 2016same root cause · shared BGP routing
IL-0273A power disruption took GitHub's Redis clusters offline, the app could not boot, and GitHub served 503s for 2 hoursGitHub · 2016same root cause · shared In-memory cache
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0255
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0255/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.