Incident Library
Applied filters Root causeHardware failureFactorMonitoring blind spotOrganisationMetaClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0411 Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checks
    Meta2024
    Symptoms

    In Meta's RSC-1 and RSC-2 research GPU clusters, certain nodes had above-average rates of job failures. After each failure the same nodes kept attracting new jobs and failing them too, while health checks and repair workflows did not flag them. Meta calls these servers lemon nodes.

    Training job failure
    Root cause
    Hardware failure
    Component
    GPU hardware and memorySlurm job scheduler
    Contributing factor
    Monitoring blind spot
    Detection

    Meta's lemon detection flags nodes from their history in per-node signals such as excl_jobid_count, xid_cnt, tickets, out_count and node failure counts, because the periodic health checks and repair workflows did not catch them.

    Remediation
    • Built lemon detection to proactively identify, isolate and replace lemon nodes in the Slurm scheduler
    • Identified 40 faulty nodes, 24 on RSC-1 and 16 on RSC-2
    Lessons

    Finding defective nodes needs historic data, and lemon node detection can improve the completion rate of large jobs by over 30%. A defective node that cannot be removed from capacity leads to restart loops that make large gang-scheduled jobs impossible to run, so recovery from random failures only works once defective nodes are reliably rotated out.

    View case
Check your cluster for this

This case involves GPU hardware and memory and Slurm job scheduler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure1 case

Component
GPU hardware and memorySlurm job scheduler
Contributing factor
Monitoring blind spot
Organisation
Meta
  1. IL-0411 Lemon nodes in Meta's RSC-1 and RSC-2 GPU clusters failed job after job without being caught by health checks
    Meta2024