Incident Library

IL-0481 Infrawaves · 2026-06-01 (published)ai-infranetworking

RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobs

Infrawaves builds and operates large GPU training clusters and runs VCCL, its collective communication library built on NCCL, in them. RNIC port downs are frequent in these clusters, and the paper says about half of the short link flaps recover within seconds. When the IP addresses were configured through NetworkManager, an RNIC port going down made the port lose its IP address for a short time. When the address returned, the port's RoCEv2 GID number had been incremented, and that change collapsed the communication collective and crashed the LLM training job.

Driver-level patches can stop the GID from being incremented, but the authors note that they bring compatibility risks in virtualised networks. Infrawaves instead changed how IP addresses are configured. Ubuntu has two basic methods, NetworkManager and the Networking service; with the Networking service the IP address stays in place when the port goes down, so there is no temporary loss and the GID number does not change. The paper adds that other Linux distributions can use a similar approach.

Severity
–
Scope
LLM training jobs: the communication collective collapsed and the training job crashed
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

In Infrawaves' RoCEv2 GPU training clusters, LLM training jobs crashed because their communication collective collapsed after an RNIC port flapped. The Global Identifier (GID) of the RDMA network card's port, which in RoCEv2 is expected to stay the same after initialisation, had changed: with the default OS network manager, NetworkManager, the port briefly lost its IP address while it was down, and the GID number was incremented even though the original IP address came back. The paper describes the GID shift as non-deterministic.

Symptom Link flappingTraining job failure

Root cause

Root cause Software defect

Component NIC, cable and switch port linkInfiniBand fabricNCCLHost routes and routing rules
Contributing factor Unreviewed defaults
Trigger
Ports of the RDMA network cards (RNICs) on the GPU servers flapped and were briefly down.

Detection

–

Remediation

Action items

–

Lessons

Host networking configuration is critical to the reliability of the collective communication library. In RoCEv2 a GID is expected to stay fixed after initialisation, but an IP configuration method that briefly drops the address when an RNIC port goes down can change it and bring down the training job. Changing the IP configuration method avoided the GID shift without the compatibility risks that driver-level patches carry in virtualised networks.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, NIC, cable and switch port link, InfiniBand fabric, NCCL among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.