Incident Library
Applied filters ComponentShared network filesystemSymptomStorage disconnected or stale mountsOrganisationCrusoe CloudClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0422 Metadata inconsistencies in Crusoe Cloud's VAST storage in eu-iceland1-a caused I/O errors and failed VM operations
    Crusoe Cloud2025
    Symptoms

    On 16 December 2025 at 17:13 UTC, Crusoe Cloud reported slowness in Shared Storage in its eu-iceland1-a region, which runs on storage from its vendor VAST. Within half an hour the scope was widened to significant slowness and intermittent access problems for Shared Storage users, affecting shared disks and GPU virtual machines in eu-iceland1.

    Storage disconnected or stale mountsThroughput degradation
    Root cause
    unknown
    Component
    Shared network filesystem
    Contributing factor
    Harmful manual action
    Remediation
    • Implemented a fix for the storage performance issues, returning latency and throughput to normal
    • Ran a targeted integrity verification with the storage cluster in a protective maintenance state
    • Lifted the broad protective maintenance restrictions after the critical phase of the verification
    • Changed the configuration to isolate background recovery processes from active customer workloads
    • Ran integrity scans and automated remediation workflows modelled on validated manual recovery procedures
    • Upgraded the VAST storage software to add automated tooling for recovery
    • Rebooted the VAST cluster to restore access to shared volumes and VM operations
    • Applied a configuration update to the storage nodes to speed up recovery workflows
    • Remediated the storage node failures of 28 December
    • Applied further VAST storage software upgrades on 28, 30 and 31 December and 1 January
    Action items
    • VAST to run a full post-incident review and add automation and safeguards that remove reliance on manual steps
    • Review the recent stability of the storage cluster with the vendor to prevent recurrences
    View case
Check your cluster for this

This case involves Shared network filesystem. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

unknown1 case

Component
Shared network filesystem
Contributing factor
Harmful manual action
Organisation
Crusoe Cloud
  1. IL-0422 Metadata inconsistencies in Crusoe Cloud's VAST storage in eu-iceland1-a caused I/O errors and failed VM operations
    Crusoe Cloud2025