Cases2 cases

1 to 2 of 2

1. IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardware

   Meta2024

   Symptoms

   Over a 54-day snapshot of Llama 3 405B pre-training the training job was interrupted 466 times. 47 interruptions were planned: automated maintenance such as firmware upgrades, or operator actions such as configuration or dataset updates. The other 419 were unexpected. Because training is synchronous, a failure on a single GPU can force a restart of the whole job.

   Training job failureGPU unavailableNCCL timeout

   Root cause

   Hardware failure

   Component

   GPU hardware and memoryNCCLPyTorch

   Contributing factor

   Single point of failure

   Detection

   The communication library's state is monitored and a stalled collective is timed out automatically; on an NCCLX watchdog or heartbeat timeout the PyTorch NCCL flight recorder dumps its tracing data, including collective metadata and stack traces.

   Remediation

   • handled almost every interruption through automation, with significant manual intervention only three times in 54 days

   Action items

   • Reduce job startup and checkpointing time
   • Use the PyTorch NCCL flight recorder to record every communication event and collective duration and to dump traces on NCCLX watchdog or heartbeat timeouts
   • Turn on heavier tracing and metadata collection in production through online configuration changes, without a code release or job restart
   • Give PyTorch access to NCCLX internal state so stalls are detected and localised faster, with a snapshot of the failing collective's finished and pending transfers

   Lessons

   Training on 16K GPUs has more failure modes than Meta's much larger CPU clusters, and because training is synchronous a single GPU failure can force a restart of the whole job. Meta raised effective training time by cutting job startup and checkpoint time and by building tools for fast diagnosis, and kept it above 90% while automated firmware and kernel maintenance caused at least one interruption a day.

   View case
2. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined

   Hans H. Kim2023

   Symptoms

   The kernel on a master node of the Kubernetes cluster logged disk I/O errors: "[HAWKEYE] kernel:: [22329228.540857] blk_update_request: I/O error, dev sda, sector 795030104 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0". The sda disk switched to read-only, and the machine needed a new disk and an operating system reinstall.

   Node NotReady

   Trigger

   A hardware failure of the disk on the first control plane node.

   Root cause

   Hardware failure

   Component

   KubesprayetcdNode root diskConfigMaps and Secrets

   Contributing factor

   Single point of failure

   Detection

   Disk I/O errors from blk_update_request on dev sda appeared in the node's kernel log.

   Remediation

   • reordered the kube_control_plane group in the Kubespray inventory so that node-2 became the first control plane node
   • removed the failed machine from the cluster
   • edited the cluster-info ConfigMap in kube-public to point at a live control plane node and updated certificate-authority-data
   • installed and configured kubelet and the other components on a new node and joined it as a control plane node with kubeadm, skipping the etcd join phase
   • added the etcd member separately

   Lessons

   In a Kubespray cluster the first node of the kube_control_plane inventory group is special, so its failure needs its own procedure: move another node into the first position, fix the cluster-info ConfigMap, join a new control plane node and add the etcd member back. etcd should be brought back to full membership before another control plane node fails.

   View case

By root cause1