Incident Library

IL-0412 Meta · 2024-10-29 (published)ai-infranetworking

InfiniBand link errors and flapping links cost Meta's RSC-1 fabric as much as 50-75% of its bandwidth during bring-up

The paper, by engineers of FAIR at Meta, analyses 11 months of data from Meta's two research clusters, RSC-1 (16k GPUs, general ML) and RSC-2 (8k GPUs, mostly vision), covering 4 million jobs and over 150 million A100 GPU hours. Both are bare-metal DGX clusters with eight A100 80GB GPUs per server, scheduled by Slurm, and the paper views failures through Slurm job states and node health checks. The servers are connected by an Ethernet front-end network for control and storage traffic and a rail-optimized InfiniBand back-end fabric for gradient exchange: two servers per rack, ten racks form a pod, and pods talk through spine switches. Node health checks every five minutes treat PCI or IB link errors as high severity, so the node is removed and its jobs rescheduled at once.

Network links degrade like servers, through changes in their physical or electrical properties, and training at scale will meet faulty links. Even with SHIELD enabled, the threshold for declaring a link down could be too conservative, and in the RSC-1 bring-up phase up to 50-75% of bandwidth was lost. Meta runs Adaptive Routing (AR) in its clusters: AR adjusts routing decisions to real-time network conditions, balances traffic across all links and lets packets avoid congested areas and unhealthy links, so the cost of bad links is spread across jobs instead of falling on the jobs mapped onto them. To show the effect, Meta injected bit errors into the fabric by modifying port registers with the mlxreg tool and ran the NCCL-Tests All-Reduce benchmark across 512 GPUs with and without AR, and AR kept much higher bandwidth under link errors. In a second test, many All-Reduce runs at once across 64 nodes in groups of two nodes (16 GPUs) showed lower performance variation and higher performance with AR.

Severity
–
Scope
RSC-1's InfiniBand back-end fabric during bring-up, with as much as 50-75% bandwidth loss; IB link errors also caused job failures on RSC-1 and RSC-2
Users affected
Researchers running ML jobs on RSC-1 and RSC-2
Region
–
Time to detect
–
Time to resolve
–

Symptoms

During the bring-up of Meta's RSC-1 research cluster, the InfiniBand back-end fabric lost as much as 50-75% of its bandwidth. The paper describes the faulty links that degrade communication across such a fabric as links with high error rates, links flapping between up and down states, permanently down links, and links under high congestion in a shared cluster.

In the job failure data, IB link errors are among the causes that contribute most to the attributed hardware failure rate on both clusters. For IB links this was dominated by a short period in early summer 2024, when a small number of offending nodes caused many IB link job failures and temporarily drove up the failure rate on both RSC-1 and RSC-2. On RSC-1 2% of IB link failures, and on RSC-2 6%, came together with GPU failures such as a GPU falling off the bus, which may point to a correlation with PCIe.

The switches had SHIELD, a self-healing feature that lets InfiniBand switches coordinate around failed links, but even with it enabled the threshold for counting a link as down could be too conservative, which led to retransmissions at the protocol level and possible network degradation. In the job logs, network faults are hard to tell apart from other causes: NCCL timeouts may be blamed on the network when the real cause is a deadlock, and networking problems produce errors across the whole stack.

Symptom Throughput degradationLink flappingTraining job failure

Root cause

Root cause Dependency failure

Component InfiniBand fabricNIC, cable and switch port link
Contributing factor –
Trigger
–

Detection

Node health checks run every five minutes and count PCI or IB link errors among the high-severity checks; the bandwidth loss was observed during the RSC-1 bring-up.

Remediation

Action items

–

Lessons

The network has to remove failures and route around them; without resilience mechanisms over 50% of bandwidth may be lost. Errors with several possible causes, such as NCCL timeouts, are easily blamed on a nearby component like the network, and networking faults spread errors across the stack.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, InfiniBand fabric, NIC, cable and switch port link among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.