Cases1 case

1 to 1 of 1

1. IL-0411 Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checks

   Meta2024

   Symptoms

   In Meta's RSC-1 and RSC-2 research GPU clusters, certain nodes had above-average rates of job failures. After each failure the same nodes kept attracting new jobs and failing them too, while health checks and repair workflows did not flag them. Meta calls these servers lemon nodes.

   Training job failure

   Root cause

   Hardware failure

   Component

   GPU hardware and memorySlurm job scheduler

   Contributing factor

   Monitoring blind spot

   Detection

   Meta's lemon detection flags nodes from their history in per-node signals such as excl_jobid_count, xid_cnt, tickets, out_count and node failure counts, because the periodic health checks and repair workflows did not catch them.

   Remediation

   • Built lemon detection to proactively identify, isolate and replace lemon nodes in the Slurm scheduler
   • Identified 40 faulty nodes, 24 on RSC-1 and 16 on RSC-2

   Lessons

   Finding defective nodes needs historic data, and lemon node detection can improve the completion rate of large jobs by over 30%. A defective node that cannot be removed from capacity leads to restart loops that make large gang-scheduled jobs impossible to run, so recovery from random failures only works once defective nodes are reliably rotated out.

   View case

By root cause1