Incident Library

IL-0422 Crusoe Cloud · 2025-12-16ai-infra

Metadata inconsistencies in Crusoe Cloud's VAST storage in eu-iceland1-a caused I/O errors and failed VM operations

Crusoe Cloud's Shared Storage in eu-iceland1-a is a VAST storage cluster. The status page reports that slowness and access problems from 16 December 2025 were caused by metadata inconsistencies in the storage and connectivity problems, without stating what caused the inconsistencies. Crusoe and VAST applied a fix for the performance issues on 16 December, ran a targeted integrity verification with the cluster in a protective maintenance state, lifted the broad restrictions early on 17 December, and later that day changed the configuration to keep background recovery processes away from active customer workloads, which stabilised performance. From 18 December the vendor ran integrity scans to map the remaining inconsistencies and built automated remediation workflows based on manual recovery procedures validated on a small subset of data, which Crusoe said it would authorise only once they were validated.

More disruption followed during recovery. A VAST storage software upgrade on 23 December introduced automated recovery tooling; later that day a manual administrative error by VAST during the data recovery process caused the 10-minute outage of all shared volumes. On 26 December two new storage network issues cut access to shared volumes, the second also stopping VM operations until VAST rebooted the cluster, and access returned early on 27 December. VAST applied a 15-minute configuration update to the storage nodes on 27 December to speed up recovery. On 28 December several storage components went offline, unrelated to the planned upgrade, and the storage node failures were remediated the same day. Further VAST software upgrades on 28, 30 and 31 December and 1 January were made to speed up recovery, all planned activities to restore storage access were complete on 2 January 2026, and the incident was resolved on 7 January 2026 after final monitoring.

Severity
–
Scope
Shared volumes (Shared Disks) in eu-iceland1-a were slow, intermittently unreachable, returned I/O errors or failed to mount; VM create, start, stop and delete operations and shared volume create, update and delete operations failed at times; all shared volumes failed reads and writes for about 10 minutes on 23 December
Users affected
–
Region
eu-iceland1-a
Time to detect
–
Time to resolve
22d 12m

Symptoms

On 16 December 2025 at 17:13 UTC, Crusoe Cloud reported slowness in Shared Storage in its eu-iceland1-a region, which runs on storage from its vendor VAST. Within half an hour the scope was widened to significant slowness and intermittent access problems for Shared Storage users, affecting shared disks and GPU virtual machines in eu-iceland1.

Crusoe named storage metadata inconsistencies as the problem, together with network connectivity that had to be stabilised. Customers saw intermittent write errors, latency spikes and brief connectivity drops. While a targeted integrity verification ran, the storage cluster went into a protective maintenance state that restricted access, and customers saw mount failures or Input/Output (I/O) errors on specific volumes, directories or files. Impact on the storage control plane then caused VM operation failures and failures to create, update or delete shared volumes.

The problem changed shape over three weeks. A fix on 16 December brought latency and throughput back to normal, but slowness remained for some customers; on 17 December the critical phase of the integrity check finished and most volumes were available again, yet some files and directories still returned I/O errors and intermittent performance drops returned while background recovery ran. On 18 December most customers were back to normal, while a limited subset stayed affected by residual metadata inconsistencies. On 23 December, from 19:52:53 to 20:03:47 UTC, read, write and update operations failed on all shared volumes for about 10 minutes. New storage network issues cut access to shared volumes on 26 December, the second also blocking VM create, start, stop and delete until VAST rebooted the cluster, and on 28 December several storage components went offline unexpectedly, again affecting shared volumes and VM management.

Symptom Storage disconnected or stale mountsThroughput degradation

Root cause

Root cause unknown

Component Shared network filesystem
Contributing factor Harmful manual action
Trigger
–

Detection

–

Remediation

Action items

Lessons

–

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Shared network filesystem among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.