Cases1 case

1 to 1 of 1

1. IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobs

   Infrawaves2026

   Symptoms

   In Infrawaves' RoCEv2 GPU training clusters, LLM training jobs crashed because their communication collective collapsed after an RNIC port flapped. The Global Identifier (GID) of the RDMA network card's port, which in RoCEv2 is expected to stay the same after initialisation, had changed: with the default OS network manager, NetworkManager, the port briefly lost its IP address while it was down, and the GID number was incremented even though the original IP address came back. The paper describes the GID shift as non-deterministic.

   Link flappingTraining job failure

   Trigger

   Ports of the RDMA network cards (RNICs) on the GPU servers flapped and were briefly down.

   Root cause

   Software defect

   Component

   NIC, cable and switch port linkInfiniBand fabricNCCLHost routes and routing rules

   Contributing factor

   Unreviewed defaults

   Remediation

   • Configured the IP addresses with the Ubuntu Networking service instead of NetworkManager, so a port down no longer removed the address or changed the GID

   Lessons

   Host networking configuration is critical to the reliability of the collective communication library. In RoCEv2 a GID is expected to stay fixed after initialisation, but an IP configuration method that briefly drops the address when an RNIC port goes down can change it and bring down the training job. Changing the IP configuration method avoided the GID shift without the compatibility risks that driver-level patches carry in virtualised networks.

   View case

By root cause1