Cases2 cases

1 to 2 of 2

1. IL-0422 Metadata inconsistencies in Crusoe Cloud's VAST storage in eu-iceland1-a caused I/O errors and failed VM operations

   Crusoe Cloud2025

   Symptoms

   On 16 December 2025 at 17:13 UTC, Crusoe Cloud reported slowness in Shared Storage in its eu-iceland1-a region, which runs on storage from its vendor VAST. Within half an hour the scope was widened to significant slowness and intermittent access problems for Shared Storage users, affecting shared disks and GPU virtual machines in eu-iceland1.

   Storage disconnected or stale mountsThroughput degradation

   Root cause

   unknown

   Component

   Shared network filesystem

   Contributing factor

   Harmful manual action

   Remediation

   • Implemented a fix for the storage performance issues, returning latency and throughput to normal
   • Ran a targeted integrity verification with the storage cluster in a protective maintenance state
   • Lifted the broad protective maintenance restrictions after the critical phase of the verification
   • Changed the configuration to isolate background recovery processes from active customer workloads
   • Ran integrity scans and automated remediation workflows modelled on validated manual recovery procedures
   • Upgraded the VAST storage software to add automated tooling for recovery
   • Rebooted the VAST cluster to restore access to shared volumes and VM operations
   • Applied a configuration update to the storage nodes to speed up recovery workflows
   • Remediated the storage node failures of 28 December
   • Applied further VAST storage software upgrades on 28, 30 and 31 December and 1 January

   Action items

   • VAST to run a full post-incident review and add automation and safeguards that remove reliance on manual steps
   • Review the recent stability of the storage cluster with the vendor to prevent recurrences

   View case
2. IL-0420 PCIe links on IBM's Vela GPU nodes dropped to a lower generation's speed and slowed training jobs by up to 2x

   IBM2025

   Symptoms

   On IBM's Vela AI cluster, GPU nodes with PCI-E Gen3 and Gen4 links sometimes performed like a much older PCI-E generation, for example at 4 GB/s, which resembles PCI-E Gen 1. Training jobs on those nodes slowed by anywhere from a few percent to 2x, and PCI-E link downgrades were the most frequently observed failure on Vela.

   Throughput degradation

   Root cause

   unknown

   Component

   PCIe link

   Contributing factor

   –

   Detection

   Autopilot measures PCI-E bandwidth between host and device on every node with periodic lightweight tests, and an alert fires when the average of 12 hourly samples falls below a threshold for the PCI-E generation; affected nodes are listed on a Grafana dashboard.

   Remediation

   • Reset the PCI-E device, usually by rebooting the VM, which resolved the downgrade at least 95% of the time
   • Rebooted VMs periodically

   Action items

   • Run periodic lightweight PCI-E link tests through Autopilot on every node and alert when the 12-hour average falls below a threshold
   • Keep investigating the root cause of the PCI-E link degradations

   Lessons

   In distributed training, a drop in the performance of a single node reduces the performance of the whole job until the slow node is identified and removed or fixed. Lightweight periodic tests averaged over a long window can find degraded PCI-E links accurately while customer jobs keep running, which lets site reliability engineers act before users report a problem.

   View case

By root cause1