Cases2 cases

1 to 2 of 2

1. IL-0412 InfiniBand link errors and flapping links cost Meta's RSC-1 fabric as much as 50-75% of its bandwidth during bring-up

   Meta2024

   Symptoms

   During the bring-up of Meta's RSC-1 research cluster, the InfiniBand back-end fabric lost as much as 50-75% of its bandwidth. The paper describes the faulty links that degrade communication across such a fabric as links with high error rates, links flapping between up and down states, permanently down links, and links under high congestion in a shared cluster.

   Throughput degradationLink flappingTraining job failure

   Root cause

   Dependency failure

   Component

   InfiniBand fabricNIC, cable and switch port link

   Contributing factor

   –

   Detection

   Node health checks run every five minutes and count PCI or IB link errors among the high-severity checks; the bandwidth loss was observed during the RSC-1 bring-up.

   Remediation

   • Enabled Adaptive Routing on the InfiniBand fabric so traffic avoids congested and unhealthy links
   • Removed nodes with IB link errors from scheduling at once through high-severity health checks and rescheduled their jobs

   Lessons

   The network has to remove failures and route around them; without resilience mechanisms over 50% of bandwidth may be lost. Errors with several possible causes, such as NCCL timeouts, are easily blamed on a nearby component like the network, and networking faults spread errors across the stack.

   View case
2. IL-0397 Orders on one exchange connection hung in pending states as its NSE leased line failed and the backup line kept flapping

   Zerodha2018

   Symptoms

   On 12 April 2018 at about 12.12 pm, the primary leased line to the NSE exchange on one of Zerodha's CTCLs went down, and the backup leased line that took over went up and down many times over the next 60 minutes. Orders sent through that CTCL were not placed on the exchange and stayed in the states open pending, validation pending and modify pending, leaving a small group of clients with hanging orders and positions.

   Link flapping

   Trigger

   The primary NSE leased line on one CTCL went down at about 12.12 pm, and the backup leased line then flapped repeatedly for about 60 minutes.

   Root cause

   Dependency failure

   Component

   NIC, cable and switch port link

   Contributing factor

   –

   Detection

   Zerodha received emails from the NSE-approved leased line vendor when the lines failed.

   Remediation

   • Shut the affected CTCL down completely so no new orders could go through it
   • Lapsed the orders left in pending states and reconciled them with the exchange

   Action items

   • Release an option to exit bracket and cover orders directly from the open position, even when no counter leg open order is shown

   View case

By root cause1