Cases1 case

1 to 1 of 1

1. IL-0412 InfiniBand link errors and flapping links cost Meta's RSC-1 fabric as much as 50-75% of its bandwidth during bring-up

   Meta2024

   Symptoms

   During the bring-up of Meta's RSC-1 research cluster, the InfiniBand back-end fabric lost as much as 50-75% of its bandwidth. The paper describes the faulty links that degrade communication across such a fabric as links with high error rates, links flapping between up and down states, permanently down links, and links under high congestion in a shared cluster.

   Throughput degradationLink flappingTraining job failure

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNIC, cable and switch port link

   Contributing factor

   –

   Detection

   Node health checks run every five minutes and count PCI or IB link errors among the high-severity checks; the bandwidth loss was observed during the RSC-1 bring-up.

   Remediation

   • Enabled Adaptive Routing on the InfiniBand fabric so traffic avoids congested and unhealthy links
   • Removed nodes with IB link errors from scheduling at once through high-severity health checks and rescheduled their jobs

   Lessons

   The network has to remove failures and route around them; without resilience mechanisms over 50% of bandwidth may be lost. Errors with several possible causes, such as NCCL timeouts, are easily blamed on a nearby component like the network, and networking faults spread errors across the stack.

   View case

By root cause1