Incident Library

IL-0475 Microsoft Azure · 2024-06-08 (published)ai-infraazurenetworking

Azure A100 InfiniBand all-reduce bandwidth dropped when over half of ToR switches' redundant uplinks were down

The SuperBench paper by Microsoft describes how hardware redundancy in Azure's AI infrastructure hides gradual degradation, which the authors call gray failure. Azure A100 clusters use over-provisioned InfiniBand networks with several redundant links between ToR switches and aggregation switches, and Microsoft found that more than half of those redundant links must work to avoid a performance regression from congestion, so at most half can be broken. Figure 3 of the paper plots the cumulative distribution of 2-node all-reduce bandwidth at different redundancy ratios for a real case on a 24-node, 192-NIC fat-tree testbed whose ToR switches each had 25% redundant uplinks. When some edge switches had over half of their redundant links down, multiple 2-node pairs running traffic together lost much of their all-reduce bus bandwidth, and the bandwidth of every pair returned to normal only after operators repaired the redundant links of all involved ToR switches to at least 50%.

The paper adds that troubleshooting usually repairs only enough redundancy to unblock the current workload, for example replacing one of several broken links, which leaves the infrastructure more likely to have the next incident when any single link fails. In Azure's A100 clusters the mean time between incidents was 17.5 hours, 38.1% of incidents took more than a day to recover, and troubleshooting started only when customers raised incidents. SuperBench, deployed in Azure for more than two years, runs benchmarks before incidents to find defective components such as these links.

Severity
–
Scope
24-node A100 testbed with 192 InfiniBand NICs: all-reduce bus bandwidth of multiple 2-node pairs dropped significantly
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

On a 24-node Azure A100 testbed with 192 InfiniBand NICs in a fat-tree topology, several 2-node pairs that ran traffic at the same time showed a significant drop in all-reduce bus bandwidth. Some edge switches had more than half of their redundant uplinks down.

Microsoft describes this as gradual degradation rather than a binary good or bad state. In Azure A100 clusters each InfiniBand top-of-rack (ToR) switch has several redundant uplinks, and when part of them break only some traffic patterns, such as all-to-all collective communication, lose throughput to congestion. Workloads that barely use the A100 GPUs or the network, such as ResNet training, do not trigger the problem early on, so monitoring the workloads already running does not reveal the lost redundancy in time.

Symptom Throughput degradation

Root cause

Root cause Dependency failure

Component InfiniBand fabric
Contributing factor Monitoring blind spot
Trigger
Some edge (ToR) switches of the testbed had more than half of their redundant uplinks to the aggregation switches down.

Detection

The drop showed in the 2-node all-reduce bandwidth measured across node pairs on the testbed. In production, Azure's troubleshooting began only when customers raised incidents.

Remediation

Action items

Lessons

Redundant links let the network keep working while degradation builds up, and only some traffic patterns slow down, so the loss is not a simple failure that monitoring of running workloads catches. Repairing just enough links to unblock the current workload does not restore the redundancy and makes the next incident more likely; Microsoft argues for stressing the hardware with standalone tests before incidents.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, InfiniBand fabric among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.