Incident Library
Applied filters ComponentObject storageSymptomTraining job failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung
    Meta AI2021
    Symptoms

    Between the team's 17 November 2021 update and 3 December 2021, the 175B-parameter OPT training run went down and had to be restarted more than 40 times, for hardware, infrastructure and training stability reasons. A hardware fault could stop the run for hours at any time of day, and several did over the Thanksgiving break.

    Training job failureGPU unavailable
    Trigger

    Nodes in the cloud training cluster failed with ECC errors, and each failure took the training run down.

    Root cause
    Dependency failure
    Component
    GPU hardware and memoryNCCLInfiniBand fabricObject storage
    Contributing factor
    No headroomLimited debug accessNo runbook
    Detection

    During these two weeks the team added monitoring of train.log, GPU burn-in testing to detect memory errors, and automated InfiniBand testing.

    Remediation
    • replaced failed nodes, and later did it through a script
    • added GPU burn-in testing to detect memory errors
    • automated InfiniBand testing
    • added monitoring of train.log
    • debugged the cloud provider's Python API until checkpoint downloads from the blob store worked well enough
    Action items
    • Run an on-call rotation for the training run, with runbooks and tooling for the most common issues, and invite more people to join it
    Lessons

    At this scale cloud hardware failures are certain, and the team concluded that automating the handling of hardware failures is necessary. Without spare nodes and burn-in tests, each failure costs hours and a replacement can bring the same fault back. The team also concluded that cloud infrastructure instability is something to always prepare for, because the customer has no control over the underlying hardware. A checkpoint large enough to need hundreds of multi-gigabyte downloads makes the storage path part of recovery time.

    View case
Check your cluster for this

This case involves GPU hardware and memory and InfiniBand fabric. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
GPU hardware and memoryInfiniBand fabricNCCLObject storage
Contributing factor
Limited debug accessNo runbookNo headroom
Organisation
Meta AI
  1. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hung
    Meta AI2021