IL-0423 Paperspace · 2024-06-16ai-infranetworking

A core switch failure in Paperspace's NY2 region cut VMs off their network file system and left them read-only

Paperspace's postmortem names a core switch failure in NY2 as the root cause. The failure caused a spike in traffic that hurt network performance, and the virtual machines, which reside on a Network File System, went into Read-Only mode when they lost network access to it. Paperspace restored network connectivity to the affected VMs by 18:20 UTC on 16 June, returned the VMs to Read-Write (RW) mode between 5:58 and 12:23 UTC on 17 June, and by 12:24 UTC the VMs and customer network access were available again. Paperspace said it is working on a network redesign and new equipment to prevent this kind of failure.

Severity: –
Scope: A large number of VMs in NY2, including Gradient notebooks and Core VMs, went read-only and unavailable; Gradient kept degraded performance during recovery
Users affected: Paperspace Gradient and Core customers in NY2
Region: NY2
Time to detect: –
Time to resolve: 1d 1h 54m

Symptoms

On 16 June 2024, from around 10:30 UTC, users could not interact with Gradient notebooks and Core VMs in Paperspace's NY2 region. A large number of virtual machines lost network access to the Network File System they resided on and went into Read-Only (RO) mode, so customers could not write to them. Network connectivity came back by 18:20 UTC, but the affected VMs stayed unavailable in RO mode until the next day, and after Core was fixed the Gradient platform still ran with degraded performance while it recovered.

Symptom Storage disconnected or stale mounts Network unreachable Service outage

Root cause

Root cause DNS or network fault

Component Shared network filesystem

Contributing factor –

Trigger: A core switch in NY2 failed, and the resulting spike in traffic degraded network performance.

Detection

–

Remediation

Restored network connectivity to the affected VMs by 18:20 UTC on 16 June
Returned the affected VMs from Read-Only to Read-Write mode between 5:58 and 12:23 UTC on 17 June

Action items

Redesign the network
Install new equipment

Lessons

–

Sources

https://status.paperspace.com/incidents/fjxf8t1x1tzf · official-status

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Shared network filesystem among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0423
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0423/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.