IL-0586 OpenAI · 2026-05-05 (published)ai-infranetworking
Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minute
OpenAI and Microsoft run their largest training clusters on a two-tier multi-plane Ethernet network that uses MRC, an RDMA transport that extends the RoCE Reliable Connection protocol and sprays the packets of each queue pair across many paths on all network planes, and static source routing with SRv6 instead of dynamic routing in the switches. Cluster A uses NVIDIA GB200 GPUs with 800 Gbps CX8 NICs, NVIDIA SP4 and Broadcom TH5 switches, and a 4 x 200 Gbps multi-plane topology, so each NIC has four 200 Gb/s ports and MRC spreads each queue pair over all four. When a NIC link drops, the NIC detects it, MRC moves its paths off the failed port, and a port state bitmap in SACK packets tells the remote endpoints to avoid that plane too, which typically takes a few seconds.
During a 50K GPU production pretraining job, an optical transceiver on a T0 switch glitched and flapped all four of its links in rapid succession. Three of the four connected nodes were active in the job. Throughput fell by about 25% over the minute of flaps and recovered to full speed immediately afterwards; MRC recovered the lost packets quickly enough that the impact on the job was relatively small; the job did not crash, no queue pair failed and no node had to be removed from the job. The authors state that such transceiver glitches are rarer than single link flaps but happen occasionally, and that losing NIC-to-T0 links affects performance in a way that T0-to-T1 link failures do not.
- Severity
- –
- Scope
- A 50K GPU production pretraining job on Cluster A ran at about 25% lower throughput for about a minute; three nodes in the job lost links
- Users affected
- –
- Region
- –
- Time to detect
- –
- Time to resolve
- 1m
Symptoms
During a 50K GPU production pretraining job at OpenAI on Cluster A, all four links of one optical transceiver on a T0 switch flapped in rapid succession, and the job's throughput dropped by about 25% for the minute of flaps. The four links went to NICs on four different nodes, three of which were running in the job.
The NICs and the T0 switch recorded the links as down at slightly different times. Packet loss rose on the nodes whose ports flapped and on the nodes that were sending to them at that moment. When a NIC link drops, many packets are lost right after the drop, and in the CX8 implementation of MRC remapping the paths of all the queue pairs in use is not instantaneous, which shows as a dip in job throughput.
Because the job is synchronous pretraining, the slowest node sets the pace for the whole job, so a few nodes losing links slowed the whole job. Throughput returned to full speed right after the flaps ended. The job did not crash, no queue pair failed and the affected nodes did not have to be removed from the job.
Root cause
Root cause Hardware failure
- Trigger
- An optical transceiver on a T0 switch had a glitch.
Detection
The NICs detected their links going down and the T0 switch recorded the same links as down; the event is shown in the job's throughput and per-node packet loss graphs.
Remediation
- MRC recovered the packets lost during the flaps quickly, so the impact on the job stayed relatively small
Action items
–
Lessons
With MRC, failures of T0-to-T1 links and T1 switches have almost no effect on training, but NIC-to-T0 links and T0 switches still have a measurable effect and need more care. Most NIC-to-T0 link losses are transient, and a job can often recover to full speed quickly without evicting nodes.
Sources
References
Related cases
IL-0478A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training stepsAmazon · 2026same root cause · shared NIC, cable and switch port link
IL-0418Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputersGoogle · 2024same root cause · shared NIC, cable and switch port link
IL-0540Korea's local government administrative services and Government24 went down as a router port dropped most large packetsKorea Ministry of the Interior and Safety · 2023same root cause · shared NIC, cable and switch port link
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0586
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0586/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.