Incident Library
Applied filters ComponentInfiniBand fabricSymptomLink flappingOrganisationMetaClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0412 InfiniBand link errors and flapping links cost Meta's RSC-1 fabric as much as 50-75% of its bandwidth during bring-up
    Meta2024
    Symptoms

    During the bring-up of Meta's RSC-1 research cluster, the InfiniBand back-end fabric lost as much as 50-75% of its bandwidth. The paper describes the faulty links that degrade communication across such a fabric as links with high error rates, links flapping between up and down states, permanently down links, and links under high congestion in a shared cluster.

    Throughput degradationLink flappingTraining job failure
    Root cause
    Dependency failure
    Component
    InfiniBand fabricNIC, cable and switch port link
    Contributing factor
    –
    Detection

    Node health checks run every five minutes and count PCI or IB link errors among the high-severity checks; the bandwidth loss was observed during the RSC-1 bring-up.

    Remediation
    • Enabled Adaptive Routing on the InfiniBand fabric so traffic avoids congested and unhealthy links
    • Removed nodes with IB link errors from scheduling at once through high-severity health checks and rescheduled their jobs
    Lessons

    The network has to remove failures and route around them; without resilience mechanisms over 50% of bandwidth may be lost. Errors with several possible causes, such as NCCL timeouts, are easily blamed on a nearby component like the network, and networking faults spread errors across the stack.

    View case
Check your cluster for this

This case involves InfiniBand fabric and NIC, cable and switch port link. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
InfiniBand fabricNIC, cable and switch port link
Contributing factor
–
Organisation
Meta
  1. IL-0412 InfiniBand link errors and flapping links cost Meta's RSC-1 fabric as much as 50-75% of its bandwidth during bring-up
    Meta2024