Cases2 cases

1 to 2 of 2

1. IL-0563 Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables

   Linode2025

   Symptoms

   At about 16:38 UTC on 17 March 2025, internal alerts fired across multiple Linode data centers because various servers had been unreachable for an extended period. At the same time customers began reporting problems with several products.

   Network unreachableService outage

   Trigger

   A release of Akamai Compute's internal backend API component was rolling out, and during the rollout the API gave inconsistent responses to the route servers.

   Root cause

   Faulty deployment

   Component

   BGP routingNetwork control plane (SDN)

   Contributing factor

   Partial rollout

   Detection

   Internal alerts fired across multiple data centers showing servers unreachable for an extended period, and customers reported problems with multiple products at the same time.

   Remediation

   • Restarted the route servers in all data centers in phases, starting with the most affected regions

   Action items

   • Hold backend API releases until there is a better way to make sure route data for route servers is contiguous

   View case
2. IL-0344 Linode's US-West data centers in Fremont and Santa Clara lost connectivity as the fibre circuits between them failed

   Linode2023

   Symptoms

   On 12 February 2023, customers in Linode's US-West region could not reach their Linodes. Santa Clara customers lost Internet connectivity after a change made at 8:45 UTC until it was reverted at 10:13 UTC, and later in the day the connection between the Fremont and Santa Clara data centers was cut completely, leaving both with degraded service or a full outage until 16:25 UTC; customers could also have trouble provisioning, powering down or rebooting services.

   Network unreachable

   Trigger

   A single link of the A-side fibre circuit between Fremont and Santa Clara went down on 11 February, the whole B-side circuit followed on 12 February, and during troubleshooting with the hardware vendor a non-intrusive command triggered a rare bug that took down all A-side links.

   Root cause

   Software defect

   Component

   Backbone network (WAN)NIC, cable and switch port link

   Contributing factor

   Overlapping failuresHarmful manual action

   Detection

   The response team began investigating when the A-side link went down on 11 February. The status page opened at 10:22 UTC on 12 February to investigate reports of connectivity issues in the US-West region.

   Remediation

   • Reverted the change that removed internet traffic from the Santa Clara border routers
   • Performed an emergency code upgrade of both circuits, restoring connectivity between the sites
   • Restored the B-side path with the hardware vendor and fiber provider, regaining full redundancy

   View case

By root cause2