IL-0563 Linode · 2025-03-17networking
Linode servers lost connectivity in all data centers as route servers built incomplete BGP route tables
Akamai Compute was rolling out a release of its internal backend API component. While the rollout was in progress, the API's responses were inconsistent between endpoints, and the route servers that read route data from it built incomplete route tables for BGP. With incomplete routes, connectivity broke in all data centers from about 16:38 UTC, and Amsterdam (NL-AMS) came close to a full outage.
When the release concluded at 17:10, the data between endpoints became consistent again and most data centers recovered by themselves. To clear the remaining impact, Linode restarted the route servers in all data centers from about 18:30, in phases with the most affected regions first, and finished at about 21:34. Backend API releases were put on hold until there is a better way to make sure the route data given to route servers is contiguous.
- Severity
- –
- Scope
- network connectivity to servers in all data centers; in Amsterdam, deploying and booting Linodes and other host-level jobs failed and the data center was nearly fully down; Object Storage and LKE access could fail
- Users affected
- –
- Region
- all regions, worst in Amsterdam (NL-AMS)
- Time to detect
- –
- Time to resolve
- 4h 56m
Symptoms
At about 16:38 UTC on 17 March 2025, internal alerts fired across multiple Linode data centers because various servers had been unreachable for an extended period. At the same time customers began reporting problems with several products.
Networking problems showed up in all data centers, and Amsterdam (NL-AMS) was hit hardest. There customers could not deploy new Linodes, boot existing Linodes or run other host-level jobs, and the data center had a near-total outage. Access to services such as Object Storage and Linode Kubernetes Engine (LKE) could also fail.
When the release behind the problem finished at 17:10 UTC, most data centers recovered on their own. The remaining impact, worst in Amsterdam, lasted until route servers were restarted region by region between about 18:30 and 21:34 UTC.
Root cause
Root cause Faulty deployment
- Trigger
- A release of Akamai Compute's internal backend API component was rolling out, and during the rollout the API gave inconsistent responses to the route servers.
Detection
Internal alerts fired across multiple data centers showing servers unreachable for an extended period, and customers reported problems with multiple products at the same time.
Remediation
- Restarted the route servers in all data centers in phases, starting with the most affected regions
Action items
- Hold backend API releases until there is a better way to make sure route data for route servers is contiguous
Lessons
–
Sources
- https://status.linode.com/incidents/78l7qy9m2sn0 · official-status · archived copy
Related cases
IL-0356Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failedRailway · 2026same root cause · shared Network control plane (SDN)
IL-0354Fly.io private networking (6PN) broke for some machines after edges received gateway /80 routesFly.io · 2026same root cause · shared Network control plane (SDN)
IL-0530Virtual router controllers in us-central1 ran out of memory in a loop, cutting networking for new and migrated VMsGoogle Cloud · 2023same root cause · shared Network control plane (SDN)
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0563
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0563/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.