Incident Library

IL-0476 ByteDance · 2025-04-26 (published)ai-infranetworking

PCIe downgrade on one machine slowed a 128-machine V100 training job for 40 minutes through a PFC surge

The Minder paper by ByteDance describes faulty-machine detection for its production training clusters, where seven months of data show about two faults a day and manual diagnosis that took over half an hour on average and sometimes days. Its example is a 128-machine training task in which one machine's PCIe link, the intra-host bus that connects devices such as GPUs, NICs and SSDs, degraded from 6.4Gbps to 4Gbps. The machine's NIC buffer filled, the resulting bottleneck in inter-host communication made it send a surge of PFC packets, and the congestion raised ECN and CNP counts and cut NIC throughput on all machines from 6.5Gbps to 4.9Gbps. With less data, GPU tensor core usage fell and the whole task ran slower for 40 minutes. The task did not stop, so no alert fired, and engineers from the training, network, storage and hardware teams needed 40 minutes to find the machine.

ByteDance built Minder to replace this manual process. It pulls per-second monitoring metrics of every machine from the data APIs, denoises each metric with its own model, and flags the machine whose metrics stay different from the other machines for a period, checking the metrics most sensitive to faults first. In the paper's table of fault types, PCIe downgrading made up 6.6% of faults over seven months and the PFC metric showed an abnormal pattern in 100% of its instances; it was 8.6% of the 150 fault instances in the evaluation dataset. Minder has run in production for over a year and reacts to faults within 3.6 seconds on average.

Severity
–
Scope
128-machine training task on thousands of V100 GPUs slowed severely for 40 minutes; NIC throughput on all machines fell from 6.5Gbps to 4.9Gbps
Users affected
–
Region
–
Time to detect
40m
Time to resolve
–

Symptoms

A distributed training task on 128 machines in ByteDance's production environment slowed down severely for 40 minutes but kept running, holding thousands of V100 GPUs at reduced speed. The alerting in place notified engineers only when a task stopped completely, so the slowdown itself raised no alert.

The NIC throughput of all machines in the task dropped from 6.5Gbps to 4.9Gbps, and GPU tensor core usage fell because less data reached the GPUs. On one machine the PFC Tx Packet Rate, the count of Priority-based Flow Control (PFC) packets sent by its RDMA devices, surged far above the other machines, whose PFC patterns had been uniform before the fault. Explicit Congestion Notifications (ECN) received and Congestion Notification Packets (CNP) sent also rose.

Finding the faulty machine took 40 minutes and several teams. The engineer in charge checked model information, parallelism settings, dependencies and environment and Megatron-LM parameters; the network team checked intra-host throughput, RDMA traffic, packet loss, congestion indicators, drivers and routing; the storage and hardware teams checked HDFS and SSD usage, GPU and CPU usage, NIC health and machine scheduling. The PFC packet rates that pointed at the machine were not inspected promptly, because the logs engineers read do not contain monitoring metrics of that kind.

Symptom Throughput degradation

Root cause

Root cause Hardware failure

Component PCIe linkInfiniBand fabric
Contributing factor No alert on limit
Trigger
The PCIe link of one machine degraded from 6.4Gbps to 4Gbps during the training task.

Detection

No alert fired because alerts came only when a task stopped. Engineers from several teams found the faulty machine by manual diagnosis, which took 40 minutes.

Remediation

–

Action items

Lessons

Some faults slow a task without stopping it, and alerting that waits for a halted task misses them. Logs alone are incomplete because they do not contain monitoring metrics such as PFC packet rates, and no single metric reveals every fault type, so ByteDance compares machines with each other on several metrics and treats a difference that lasts for minutes as a fault.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, PCIe link, InfiniBand fabric among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.