Incident Library
Applied filters ComponentBGP routingFactorSignal buried in noiseClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020
    Symptoms

    At 11:59AM CDT on Friday 28 August 2020, Basecamp's Ops team received a flood of alerts from PagerDuty, Nagios and Prometheus. Monitoring showed that the newer of the two metro links between the data centers in Ashburn, VA and Chicago, IL had been lost for about 20-30 seconds, and BC3 kept responding slowly after the link came back.

    Link flappingHigh latencyReplication lag
    Trigger

    Both redundant metro links between Basecamp's Ashburn and Chicago data centers dropped for short periods; monitoring showed the newer link down for about 20-30 seconds.

    Root cause
    Dependency failure
    Component
    Backbone network (WAN)BGP routingIn-memory cacheDatabase replication
    Contributing factor
    Single point of failureRetry stormSignal buried in noise
    Detection

    A flood of alerts from PagerDuty, Nagios and Prometheus, with monitoring showing the lost metro link. A lingering alert for one Redis DB instance later pointed to the replication problem.

    Remediation
    • Turned chat off at the load balancer layer so the rest of BC3 could stabilise
    • Restarted Redis on the replica that was stuck in the resynchronization loop
    Action items
    • Make the documented Redis configuration changes for the known replication issue
    • Make concrete plans to remove the shared physical path under the two metro links
    • Prove the failover systems in production at peak load
    • Make explicit room for post-incident follow-up work and push other work out
    Lessons

    Planned redundancy and live testing had given Basecamp a false sense of security, especially at peak load: moving between redundant network links or shifting load across data centers should work without issue, and Basecamp would rather find out in a controlled test with the full team ready. The follow-up from an outage needs urgency and explicitly cleared time, not just a place on a list.

    View case
  2. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020
    Symptoms

    On 16 July 2020 between 13:26 and 13:48 UTC, DNS queries to two nodes of Gandi's LiveDNS platform failed with a timeout, and retries against the other name servers of a domain failed as well. The status page reported impact for FR customers.

    DNS resolution failures
    Trigger

    A software bug in Gandi's DNS server software triggered a fault and stopped the server on two LiveDNS nodes.

    Root cause
    Software defect
    Component
    Authoritative DNS servers and recordsBGP routing
    Contributing factor
    Single point of failureSignal buried in noiseOverlapping failures
    Detection

    Monitoring alerts fired on the DNS servers, but they were first read as false positives among the noise from an earlier internal network incident.

    Action items
    • Fix the way a node is disabled when its DNS server is not working correctly
    • Fix the node configuration and add monitoring so that no node announces several clusters
    • Rework monitoring and internal training and organisation
    • Deploy the fix for the bug in the DNS server software
    View case
Check your cluster for this

These 2 cases involve BGP routing and Backbone network (WAN). kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Dependency failure1 case

Component
Backbone network (WAN)BGP routingIn-memory cacheDatabase replication
Contributing factor
Signal buried in noiseRetry stormSingle point of failure
Organisation
37signals
  1. IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat
    37signals2020

Software defect1 case

Component
BGP routingAuthoritative DNS servers and records
Contributing factor
Signal buried in noiseOverlapping failuresSingle point of failure
Organisation
Gandi
  1. IL-0359 Queries to Gandi LiveDNS timed out when two crashed anycast DNS nodes kept announcing all three name server addresses
    Gandi2020