Cases1 case

1 to 1 of 1

1. IL-0477 Slow compute on 8 ranks pushed a 10,000-GPU MoE job from 30 s to over 90 s per step; an alert pointed at the network

   Tencent2026

   Symptoms

   In an MoE training job of about 10,000 GPUs on Tencent's production cluster, iteration time rose from about 30 s to over 90 s and the ARGUS iteration-level detector raised an alert. At the same time the out-of-band monitoring system reported a "server port down" event on one of the affected nodes.

   Throughput degradation

   Root cause

   Hardware failure

   Component

   –

   Contributing factor

   Unverified assumption

   Detection

   ARGUS level L1 detected the iteration-time regression and raised an alert, and level L2's cross-rank comparison of phase durations located the 8 slow ranks.

   Remediation

   • Replaced the affected nodes, after which training speed recovered immediately

   Lessons

   Out-of-band infrastructure metrics can point at the wrong component: a communication anomaly or port event may be a secondary effect of compute degradation that propagates through collectives. Fine-grained tracing that separates compute operators from communication is what showed the network explanation was wrong.

   View case

By root cause1