Incident Library

IL-0477 Tencent · 2026-07-08 (published)ai-infra

Slow compute on 8 ranks pushed a 10,000-GPU MoE job from 30 s to over 90 s per step; an alert pointed at the network

ARGUS is Tencent's always-on tracing and diagnosis system for training on a production cluster of more than 10,000 GPUs, deployed there for over six months. It traces CPU call stacks, framework semantics and GPU kernel execution with less than 2% overhead and diagnoses in levels: L1 finds anomalous windows in each rank's iteration time, L2 compares phase durations among ranks with the same parallel role using the coefficient of variation and z-scores, L3 compares kernel duration distributions, and L4/L5 are deep dives in Grafana and Perfetto on the few suspect ranks.

Case 5 in the paper is an MoE training job of about 10,000 GPUs with TP=1, PP=9 and EP=32 whose iteration time rose from about 30 s to over 90 s. L2 pointed at 8 ranks whose compute ran about 5.7 times slower than the rest of their group, and the paper confirms compute degradation on those ranks as the root cause; the wait spread first through the EP dispatch and then through pipeline stage dependencies to the other stages of the same pipeline group. A "server port down" event from out-of-band monitoring on one affected node led operations to plan a network repair, but the slowdown showed only in compute operators. Training speed recovered immediately after the affected nodes were replaced. The paper concludes that communication anomalies seen at the infrastructure level can be a secondary effect of slow compute propagating through collectives, and that coarse metrics alone can lead to an ineffective fix.

Severity
–
Scope
MoE training job of about 10,000 GPUs: iteration time rose from about 30 s to over 90 s
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

In an MoE training job of about 10,000 GPUs on Tencent's production cluster, iteration time rose from about 30 s to over 90 s and the ARGUS iteration-level detector raised an alert. At the same time the out-of-band monitoring system reported a "server port down" event on one of the affected nodes.

ARGUS compared phase durations across the ranks of the same data-parallel group. The compute-only mlp phase on ranks 10352–10359 (PP=7, DP=272–279) had a coefficient of variation above 0.8, with durations about 5.7 times the group mean. On the Grafana heatmap of per-rank maximum operator duration, most ranks ran mlp in about 30–50 ms while those 8 ranks took 170–280 ms.

The delay spread through collective dependencies. The 8 slow ranks were in an expert-parallel (EP) group of 32 ranks (10336–10367), and the EP dispatch needs all 32 ranks, so the whole group entered the data-parallel ReduceScatter late. Because the other EP groups were already waiting, this group's ReduceScatter looked shorter than its neighbours', and pipeline stage dependencies gave ranks in every PP stage at DP=256–287 the same shorter ReduceScatter.

Operations first took the port-down event as the root cause and scheduled a network repair. ARGUS tracing showed that the anomaly appeared only in pure-compute operators with no communication, which contradicted the network explanation.

Symptom Throughput degradation

Root cause

Root cause Hardware failure

Component –
Contributing factor Unverified assumption
Trigger
–

Detection

ARGUS level L1 detected the iteration-time regression and raised an alert, and level L2's cross-rank comparison of phase durations located the 8 slow ranks.

Remediation

Action items

–

Lessons

Out-of-band infrastructure metrics can point at the wrong component: a communication anomaly or port event may be a secondary effect of compute degradation that propagates through collectives. Fine-grained tracing that separates compute operators from communication is what showed the network explanation was wrong.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.