IL-0344 Linode · 2023-02-11networking

Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed

Linode's Fremont and Santa Clara data centers in the US-West region are linked by a geographically diverse, redundant pair of multi-fiber circuits, the A-side and the B-side, which connect border routers and core switches at the two sites and carry peer Linode traffic and shared internet transit. On 11 February 2023 one A-side link failed, and early on 12 February the whole B-side circuit failed, leaving traffic on the A-side core switch links with reduced performance. An attempt at 8:45 UTC to stabilise the network by removing internet traffic from the Santa Clara border routers took Santa Clara offline until it was reverted at 10:13 UTC.

When the cause of the A-side link failure could not be found, the issue was escalated to the hardware vendor. During the vendor's troubleshooting, a non-intrusive command triggered a rare bug that failed all A-side links and severed the two data centers. The vendor advised that a code upgrade of both circuits was needed; the emergency upgrades were completed at 16:25 UTC, and traffic was back on the A-side path by 17:29 UTC. The B-side was restored at 18:30 UTC on 14 February after work with the hardware vendor and the fiber provider; the fault was a unidirectional loss of light on an intermediate segment of the circuit. The postmortem says another outage hit the two data centers on 21 February but does not describe it.

Severity: –
Scope: Internet connectivity for customers in the Fremont and Santa Clara data centers and connectivity between the two sites; customers could not reach their Linodes and could have trouble provisioning, powering down or rebooting services
Users affected: –
Region: US-West (Fremont and Santa Clara)
Time to detect: –
Time to resolve: 23h 44m

Symptoms

On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

The trouble built up over several hours. At 17:45 UTC on 11 February one link in the A-side multi-fiber bundle between the two sites went down, and internet traffic moved to the B-side with no visible customer impact. At 2:45 UTC on 12 February the whole B-side circuit went down, isolating the border routers of the two sites from each other; traffic still flowed over the A-side links between the core switches, but network performance was suboptimal.

Two actions taken during the response caused the outages. Removing all internet traffic from the Santa Clara border routers at 8:45 UTC cut Internet connectivity for Santa Clara customers, although this was not seen right away, and the change was reverted at 10:13 UTC. While troubleshooting the failed A-side link with the hardware vendor, a non-intrusive command set off a rare bug that brought down every A-side link.

Symptom Network unreachable

Root cause

Root cause Software defect

Component Backbone network (WAN)NIC, cable and switch port link

Contributing factor Overlapping failures Harmful manual action

Trigger: A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

Detection

The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

Remediation

Reverted the change that removed internet traffic from the Santa Clara border routers
Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy

Action items

–

Lessons

–

Sources

https://status.linode.com/incidents/gh22stjx5xw6 · official-status

Related cases

IL-0481 RoCEv2 GIDs changed after RNIC port flaps, collapsing the communication collective and crashing LLM training jobsInfrawaves · 2026
same root cause · shared NIC, cable and switch port link
IL-0256 Basecamp was fully down twice in two days as hardware load balancers crashed and routers lost sync after failover37signals · 2020
same root cause · shared Backbone network (WAN)
IL-0222 Google Cloud US regions had heavy packet loss and some went unreachable after the network control plane was descheduledGoogle Cloud · 2019
same root cause · shared Backbone network (WAN)
IL-0263 GitHub was slow and erroring all day and down for 18 minutes as links between access and aggregation switches saturatedGitHub · 2012
same root cause · shared NIC, cable and switch port link

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Backbone network (WAN), NIC, cable and switch port link among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0344
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0344/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.