Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checks

Symptoms

In Meta's RSC-1 and RSC-2 research GPU clusters, certain nodes had above-average rates of job failures. After each failure the same nodes kept attracting new jobs and failing them too, while health checks and repair workflows did not flag them. Meta calls these servers lemon nodes.

The paper ties lemon nodes to NODE_FAIL, the catch-all status Slurm gives through its heartbeats when a node stops responding to the health checks running on it, and names NODE_FAIL as one of the most important factors behind training job failures. Large jobs were hit hardest: before lemon detection, 14% of jobs on 512 or more GPUs failed.

Before the detection was automated, researchers excluded nodes by hand when those nodes had made their jobs fail, based on past experience. That did not scale, and excluding nodes too aggressively risked starving the cluster of capacity. The count of distinct jobs that had excluded a node (excl_jobid_count) did not correlate strongly with node failures, even though a large number of nodes had been excluded by at least one job. Finding a lemon node takes failure history over a long period before the pattern is statistically significant.

Symptom Training job failure

Root cause

Trigger

–

Detection

Meta's lemon detection flags nodes from their history in per-node signals such as excl_jobid_count, xid_cnt, tickets, out_count and node failure counts, because the periodic health checks and repair workflows did not catch them.

Remediation

• Built lemon detection to proactively identify, isolate and replace lemon nodes in the Slurm scheduler
• Identified 40 faulty nodes, 24 on RSC-1 and 16 on RSC-2

Action items

–

Lessons

Finding defective nodes needs historic data, and lemon node detection can improve the completion rate of large jobs by over 30%. A defective node that cannot be removed from capacity leads to restart loops that make large gang-scheduled jobs impossible to run, so recovery from random failures only works once defective nodes are reliably rotated out.

Sources

• https://arxiv.org/abs/2410.21680 · paper · archived copy

Related cases

• IL-0427 A GPU with uncorrectable ECC errors crashed tr1-13B training, and Slurm's next job went to the same node and crashed tooBigScience · 2021

  same root cause · shared GPU hardware and memory, Slurm job scheduler

• IL-0416 Defective GPUs silently corrupted ByteDance's LLM training, showing as loss spikes, NaN and misleading code errorsByteDance · 2026

  same root cause · shared GPU hardware and memory

• IL-0415 GSP, fallen-off-the-bus and NVLink errors left A100 GPUs on NCSA's Delta inoperable and failed the jobs running on themNCSA · 2025

  same root cause · shared GPU hardware and memory

• IL-0482 Bursts of uncorrectable A100 memory errors on Perlmutter and Polaris reached over 1,000 double-bit errors a dayNERSC · 2025

  same root cause · shared GPU hardware and memory

• IL-0421 Errors, mostly GPU faults, crashed Alibaba LLM training jobs as NCCL errors; a job lost 31.19% of its time in June 2023Alibaba Group · 2025

  same root cause · shared GPU hardware and memory

• IL-0414 Uncorrectable memory errors on NCSA Delta's H100 GPUs failed user jobs, at a 3.2× lower per-GPU MTBE than on A100NCSA · 2025

  same root cause · shared GPU hardware and memory

• IL-0238 Xid 74 NVLink errors led the GPU faults and InfiniBand flash cuts failed tasks on DeepSeek's 10,000-GPU A100 clusterDeepSeek · 2024

  same root cause · shared GPU hardware and memory

• IL-0127 Llama 3 405B pre-training on 16K H100 GPUs was interrupted 419 times unexpectedly in 54 days, mostly by faulty GPUs, HBM3 memory and other hardwareMeta · 2024

  same root cause · shared GPU hardware and memory

• IL-0239 LLM pretraining jobs on Shanghai AI Laboratory's A100 clusters kept failing midway, mostly from NVLink and GPU errorsShanghai AI Laboratory · 2024

  same root cause · shared GPU hardware and memory

• IL-0125 A GPU died about once a week during a 176B-parameter training run, and sometimes the job hung instead of exiting, so SLURM never restarted itBigScience · 2022

  same root cause · shared Slurm job scheduler

• IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadableTsinghua University · 2022

  same root cause · shared GPU hardware and memory

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0411
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0411/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.