Incident Library

Organisation

Infrawaves

Infrawaves · infrawaves

1 case · 2026

Root cause Software defect
Component Host routes and routing rulesInfiniBand fabricNCCLNIC, cable and switch port link
Contributing factor Unreviewed defaults
Symptom Link flappingTraining job failure

Cases

  1. IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobsInfrawaves · 2026

    Root cause Software defect