Cases3 cases

1 to 3 of 3

1. IL-0477 Slow compute on 8 ranks pushed a 10,000-GPU MoE job from 30 s to over 90 s per step; an alert pointed at the network

   Tencent2026

   Symptoms

   In an MoE training job of about 10,000 GPUs on Tencent's production cluster, iteration time rose from about 30 s to over 90 s and the ARGUS iteration-level detector raised an alert. At the same time the out-of-band monitoring system reported a "server port down" event on one of the affected nodes.

   Throughput degradation

   Root cause

   Hardware failure

   Component

   –

   Contributing factor

   Unverified assumption

   Detection

   ARGUS level L1 detected the iteration-time regression and raised an alert, and level L2's cross-rank comparison of phase durations located the 8 slow ranks.

   Remediation

   • Replaced the affected nodes, after which training speed recovered immediately

   Lessons

   Out-of-band infrastructure metrics can point at the wrong component: a communication anomaly or port event may be a secondary effect of compute degradation that propagates through collectives. Fine-grained tracing that separates compute operators from communication is what showed the network explanation was wrong.

   View case
2. IL-0421 Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023

   Alibaba Group2025

   Symptoms

   Large LLM training jobs on Alibaba's AI clusters crashed again and again and had to restart from the last valid checkpoint. One of the largest jobs, on 4,096 GPUs in a newly deployed cluster, crashed 40 times in one month, and most crashes reached users only as "NCCL Errors".

   Training job failureNCCL timeoutGPU unavailable

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLPyTorch

   Contributing factor

   Monitoring blind spotUnverified assumption

   Detection

   Before C4D, users found crashes through NCCL errors and could stay unaware of a stalled job for up to 30 minutes until the PyTorch elastic agent killed it; C4D later detects hangs and slowdowns from collective communication timings within tens of seconds.

   Remediation

   • Allocated 64 backup GPUs on 8 servers for every 1024 GPUs on 128 servers, so that parallel training on any 128 servers of the 136-server pool keeps the same communication and performance
   • Deployed C4D to detect faulty nodes automatically, isolate them and restart the job from the most recent valid checkpoint
   • Saved checkpoints more often, every 10 minutes, to cut the work lost after each crash
   • Enhanced cooling with higher fan speeds and better air conditioning to keep GPUs from overheating
   • Identified and enhanced the most vulnerable components of the system, lowering the average error rate 3.33 times

   Action items

   • Take load variation into account in C4D for more precise slow detection

   Lessons

   Most crash causes are confined to specific nodes or devices, so isolating the faulty component quickly and restarting, while leaving root cause analysis for offline work, removes most of the error-induced downtime. Because synchronous training makes many different faults look like the same communication error, diagnosis has to use the regular rhythm of the workers at synchronisation points rather than the error message, and frequent checkpoints matter because error rates in large clusters are high.

   View case
3. IL-0219 GoCardless's API and Dashboard failed every request for 1 hour 50 minutes when Pacemaker did not promote a replica

   GoCardless2017

   Symptoms

   At 15:09 BST on 10 October 2017, GoCardless's monitoring detected a total outage of its API and Dashboard: every request failed and returned an error. By 15:11 engineers saw evidence of a disk array failure on the primary Postgres node, but the Pacemaker cluster had not failed over to the synchronous replica as it should have.

   Service outageFailover failure

   Trigger

   The RAID controller of the primary Postgres node lost 3 disks of its array at once, so all reads and writes on the primary failed.

   Root cause

   Hardware failure

   Component

   DatabaseDatabase failover manager

   Contributing factor

   Overlapping failuresUnverified assumption

   Detection

   GoCardless's monitoring detected the total outage of the API and Dashboard at 15:09 BST, and engineers began investigating.

   Remediation

   • Powered off the broken primary Postgres node
   • Cleared the Pacemaker error counts with crm resource cleanup, which had no effect
   • Edited recovery.conf on the synchronous replica under maintenance mode several times, which Pacemaker reverted
   • Put the cluster into maintenance mode, configured the synchronous replica as primary and started Postgres by hand
   • Reconfigured the backend applications to connect to the new primary's actual IP address
   • Bootstrapped a third node as an asynchronous replica
   • Added a VIP managed by hand by the infrastructure team for manual failover
   • Verified data integrity and found no corruption
   • Migrated to a new Pacemaker-managed database cluster in a maintenance window and decommissioned the old one

   Action items

   • Express the Backup VIP rule as an INF preference for a replica instead of a -INF preference against the primary
   • Replace the VIPs with proxies on the application servers that route traffic by the state of the cluster
   • Run harsher fault injection tests of the Postgres cluster and consider automation that injects failures continually
   • Consider adding arbitrary restrictions to game day exercises, such as automation that has failed and a problem that cannot be diagnosed, so engineers practise bringing the service back another way

   Lessons

   A rule that two resources must not run together does not behave like the opposite of a rule that they must run together, and in some failure conditions the difference stopped Pacemaker from promoting a replica. A misconfiguration that shows only when two processes crash at almost the same time will not appear in basic tests or daily operations, only through fault injection. After two years in which the automation had handled every failure, the team's skill at controlling the cluster by hand had faded, and a runbook unused for years would likely have been out of date.

   View case

By root cause1