Incident Library

Organisation

Meta AI

Meta AI · meta-ai

1 case · 2021

Root cause Dependency failure
Component GPU hardware and memoryInfiniBand fabricNCCLObject storage
Contributing factor Limited debug accessNo runbookNo headroom
Symptom GPU unavailableTraining job failure

Cases

  1. IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021

    Root cause Dependency failure