Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic

Symptoms

At about 08:49 UTC on 2 June 2019, Packet's internal network monitoring alerted on serious reachability and performance problems in several of Packet's global datacenter locations. The first investigation found routing loops and traffic being blackholed to specific internal and Internet destinations.

The damage was uneven. Only traffic between particular, seemingly random pairs of source and destination IP addresses was hit while other customer traffic flowed normally, and delivery from Packet's US datacenters to broadband ISPs suffered the most. Server interface connectivity was never affected. Customers first lost upstream connectivity briefly in some Packet Cloud regions, then for the rest of the outage had poor routing with high latency and some paths that could not be reached at all.

Provisioning was affected too: from 09:18 engineers investigated provisioning and deprovisioning problems in the NRT1 availability zone, and at 10:11 found SJC1 device provisioning affected as well. The routing issue was identified at 10:13, and at 12:10 network engineers confirmed it came from an outage at Packet's upstream network provider Zayo. Even after Zayo was taken out of Packet's provider mix, some large broadband and cellular networks that also depend on Zayo kept degraded connectivity to Packet and to other destinations.

Symptom Network unreachablePacket lossHigh latency

Root cause

Trigger

A trans-continental US backbone link in the network of Packet's upstream provider Zayo had a hardware failure and blackholed IP traffic routed over it.

Detection

Packet's internal network monitoring alerted at 08:49 UTC on reachability issues in several global datacenter locations, and engineers confirmed them at 08:52.

Remediation

• Migrated traffic to an alternative network and removed Zayo from the provider mix
• Applied route filters to return service to normal
• Optimised specific routing policies to improve latency across the backbone with the new traffic mix

Action items

• Expand internal connectivity monitoring and alerting of on-call operations teams
• Expand network automation and routing so traffic can be balanced across all available backbone paths and an operator can trigger automated re-routes across all Packet datacenters in a major provider incident
• Make Packet's provisioning systems less dependent on third party network providers and inter-datacenter connectivity

Lessons

–

Sources

• https://status.equinixmetal.com/incidents/tp130h5m6px0 · official-status

Related cases

• IL-0267 Wikimedia's eqsin cache site went unreachable when both of its transport links to the core data centers were downWikimedia Foundation · 2023

  same root cause · shared BGP routing, Backbone network (WAN)

• IL-0255 Basecamp's two redundant links between data centers dropped together and a Redis replica resync loop stalled chat37signals · 2020

  same root cause · shared BGP routing, Backbone network (WAN)

• IL-0509 SourceHut was down for 170 hours after a DDoS led its upstream network to null-route the PHL datacenter's subnetSourceHut · 2024

  same root cause · shared BGP routing

• IL-0258 FastMail was unreachable for almost two hours when the GRE tunnels carrying its traffic from Level 3 went downFastmail · 2016

  same root cause · shared BGP routing

Check your cluster for this

For AI tools and scripts

• This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0358
• The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0358/similar
• MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.