Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases8 cases

7 to 8 of 8

  1. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts
    Linode2022
    Symptoms

    In the week of 10 October 2022, Linode Object Storage in Newark had two incidents of reduced performance and availability, on 10 and 16 October, and in some cases the front-end gateways returned 504 gateway timeouts to customers. The second incident began while the first was still being investigated.

    Intermittent timeoutsThroughput degradation
    Trigger

    One back-end Object Storage node had a hardware problem that was not an outright failure, so the node kept going up and down.

    Root cause
    Hardware failure
    Component
    Object storage
    Contributing factor
    Monitoring blind spotAutomation amplification
    Remediation
    • Removed the failed back-end infrastructure from the Object Storage cluster
    Action items
    • Improve detection so subtle problems with failing or misbehaving hardware are found and triaged before they affect the wider cluster
    • Describe Ceph-related events more accurately to on-call administrators and define alerts more clearly
    Lessons

    Hardware that misbehaves without failing outright can cause service-wide problems, and Linode's internal monitoring had gaps in spotting such subtle issues before they affected the wider cluster.

    View case
  2. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable
    Tsinghua University2022
    Symptoms

    The GLM-130B pretraining run on 96 nodes, started on 2022-05-06 at 04:00, was interrupted again and again by single nodes that broke down, lost connection or reported missing GPUs, and each time training had to be restarted from a saved step. On 2022-05-30 a restart got stuck in data loading because of a Lustre file system error, and the 2.3T text corpora could no longer be read.

    Training job failureNode NotReadyGPU unavailableStorage disconnected or stale mountsThroughput degradation
    Trigger

    Individual nodes of the 96-node allocation failed during training, through breakdowns, a lost connection, missing GPUs, a CPU error and an NVLink error, and on 2022-05-30 the Lustre file system that held the training corpus failed.

    Root cause
    Hardware failure
    Component
    GPU hardware and memoryShared network filesystem
    Contributing factor
    –
    Detection

    The Lustre error was found when a restart from global_step23200 stayed stuck in data loading.

    Remediation
    • Restarted the training from the last saved step after each node failure
    • Changed the checkpoint saving interval from 500 steps to 100 steps
    • Switched the job to another node pool and restarted from step 15600
    • Removed the suspected replacement node n30076 from step 16600, which did not change the speed
    • Copied the 2.3T text corpora from a backup disk back to the Lustre file system, which took a few days
    View case
Check your cluster for this

These 8 cases involve NIC, cable and switch port link and GPU hardware and memory. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Hardware failure8 cases

Component
NIC, cable and switch port link3GPU hardware and memory1InfiniBand fabric1NCCL1Object storage1PCIe link1Server power supply and power capping1Shared network filesystem1TPU1
Contributing factor
Monitoring blind spot2Unverified assumption1Automation amplification1No alert on limit1
Organisation
Amazon1ByteDance1Google1IBM1Linode1OpenAI1Tencent1Tsinghua University1
  1. IL-0477 Slow compute on 8 ranks pushed a 10,000-GPU MoE job from 30 s to over 90 s per step; an alert pointed at the network
    Tencent2026
  2. IL-0478 A failed GPU network adapter made NCCL silently reroute through GPU 0's adapter and slowed MoE training steps
    Amazon2026
  3. IL-0586 Four NIC-to-T0 links flapped in rapid succession and cut a 50K-GPU pretraining job's throughput about 25% for a minute
    OpenAI2026
  4. IL-0476 PCIe downgrade on one machine slowed a 128-machine V100 training job for 40 minutes through a PFC surge
    ByteDance2025
  5. IL-0419 A Granite-20B training job on IBM's Vela ran almost 3x slower because one node's power card fault throttled its GPUs
    IBM2025
  6. IL-0418 Machine, ICI link and optical circuit switch failures interrupted training jobs on Google's TPU v4 supercomputers
    Google2024
  7. IL-0381 A misbehaving back-end node in Linode's Newark Object Storage slowed the service and caused 504 gateway timeouts
    Linode2022
  8. IL-0429 GLM-130B training on 96 nodes kept stopping on node failures, and a Lustre error left the 2.3T corpus unreadable
    Tsinghua University2022