Cases1 case

1 to 1 of 1

1. IL-0586 Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minute

   OpenAI2026

   Symptoms

   During a 50K GPU production pretraining job at OpenAI on Cluster A, all four links of one optical transceiver on a T0 switch flapped in rapid succession, and the job's throughput dropped by about 25% for the minute of flaps. The four links went to NICs on four different nodes, three of which were running in the job.

   Link flappingThroughput degradation

   Trigger

   An optical transceiver on a T0 switch had a glitch.

   Root cause

   Hardware failure

   Component

   NIC, cable and switch port link

   Contributing factor

   –

   Detection

   The NICs detected their links going down and the T0 switch recorded the same links as down; the event is shown in the job's throughput and per-node packet loss graphs.

   Remediation

   • MRC recovered the packets lost during the flaps quickly, so the impact on the job stayed relatively small

   Lessons

   With MRC, failures of T0-to-T1 links and T1 switches have almost no effect on training, but NIC-to-T0 links and T0 switches still have a measurable effect and need more care. Most NIC-to-T0 link losses are transient, and a job can often recover to full speed quickly without evicting nodes.

   View case

By root cause1