Incident Library

Organisation

Imbue

Imbue · imbue

1 case · 2024

Root cause Dependency failure
Component GPU driverInfiniBand fabricNCCLPyTorch
Contributing factor Unverified assumptionMonitoring blind spotPartial rollout
Symptom GPU unavailableLink flappingNCCL timeoutThroughput degradationTraining job failure

Cases

  1. IL-0025 Failed boots, InfiniBand link flapping, NCCL hangs and throughput sags while bringing up a 4,088-GPU H100 training clusterImbue · 2024

    Root cause Dependency failure