Incident Library
Applied filters Root causeSoftware defectSymptomLink flappingClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobs
    Infrawaves2026
    Symptoms

    In Infrawaves' RoCEv2 GPU training clusters, LLM training jobs crashed because their communication collective collapsed after an RNIC port flapped. The Global Identifier (GID) of the RDMA network card's port, which in RoCEv2 is expected to stay the same after initialisation, had changed: with the default OS network manager, NetworkManager, the port briefly lost its IP address while it was down, and the GID number was incremented even though the original IP address came back. The paper describes the GID shift as non-deterministic.

    Link flappingTraining job failure
    Trigger

    Ports of the RDMA network cards (RNICs) on the GPU servers flapped and were briefly down.

    Root cause
    Software defect
    Component
    NIC, cable and switch port linkInfiniBand fabricNCCLHost routes and routing rules
    Contributing factor
    Unreviewed defaults
    Remediation
    • Configured the IP addresses with the Ubuntu Networking service instead of NetworkManager, so a port down no longer removed the address or changed the GID
    Lessons

    Host networking configuration is critical to the reliability of the collective communication library. In RoCEv2 a GID is expected to stay fixed after initialisation, but an IP configuration method that briefly drops the address when an RNIC port goes down can change it and bring down the training job. Changing the IP configuration method avoided the GID shift without the compatibility risks that driver-level patches carry in virtualised networks.

    View case
  2. IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover
    37signals2020
    Symptoms

    On Tuesday 1 September 2020 Basecamp could not be loaded at all for 17 minutes, from 9:51am to 10:08am Central Time (14:51–15:08 UTC), and nothing seemed to work. On Wednesday 2 September it happened again for 15 minutes, from 10:50am to 11:05am Central Time (15:50–16:05 UTC).

    Service outageLink flappingHigh latency
    Trigger

    On 1 September the network links between the data centers went offline, and during the recovery a hardware load balancer crashed. On 2 September the primary load balancer in Virginia crashed again, and later its secondary crashed too.

    Root cause
    Software defect
    Component
    On-premises load balancerBackbone network (WAN)
    Contributing factor
    Overlapping failuresNo headroom
    Remediation
    • On 1 September, intervened by hand to get the network routers working with the standby load balancer
    • On 2 September, fixed the same router and load balancer desynchronization by hand
    Action items
    • Find the cause of the load balancer crash with the vendor and apply its bugfix
    • Replace the hardware load balancers
    • Reroute the redundant cross-datacenter network paths for proper circuit diversity
    • Evaluate moving from hardware to software load balancers, which can be deployed in minutes
    • Evaluate decentralizing the load balancer architecture to limit the impact of one failure
    • Assess the systems for resilience, fragility and risk, and review the assessment process itself
    Lessons

    Basecamp plans for failures like these and practises responses to catastrophic failures in its live systems, but three obscure issues in five days chained into overlapping failures. What caught the team by surprise were cascading failures that exposed unexpected fragility and hard paths to recovery, which Basecamp says it can prepare for.

    View case
Check your cluster for this

These 2 cases involve Backbone network (WAN) and Host routes and routing rules. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Backbone network (WAN)1Host routes and routing rules1InfiniBand fabric1NCCL1NIC, cable and switch port link1On-premises load balancer1
Contributing factor
Unreviewed defaults1No headroom1Overlapping failures1
Organisation
37signals1Infrawaves1
  1. IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobs
    Infrawaves2026
  2. IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover
    37signals2020