Cases12 cases

11 to 12 of 12

1. IL-0358 Packet Public Cloud locations had high latency and unreachable paths from routing loops and blackholed traffic

   Packet2019

   Symptoms

   At about 08:49 UTC on 2 June 2019, Packet's internal network monitoring alerted on serious reachability and performance problems in several of Packet's global datacenter locations. The first investigation found routing loops and traffic being blackholed to specific internal and Internet destinations.

   Network unreachablePacket lossHigh latency

   Trigger

   A trans-continental US backbone link in the network of Packet's upstream provider Zayo had a hardware failure and blackholed IP traffic routed over it.

   Root cause

   Dependency failure

   Component

   Backbone network (WAN)BGP routingNetwork control plane (SDN)

   Contributing factor

   –

   Detection

   Packet's internal network monitoring alerted at 08:49 UTC on reachability issues in several global datacenter locations, and engineers confirmed them at 08:52.

   Remediation

   • Migrated traffic to an alternative network and removed Zayo from the provider mix
   • Applied route filters to return service to normal
   • Optimised specific routing policies to improve latency across the backbone with the new traffic mix

   Action items

   • Expand internal connectivity monitoring and alerting of on-call operations teams
   • Expand network automation and routing so traffic can be balanced across all available backbone paths and an operator can trigger automated re-routes across all Packet datacenters in a major provider incident
   • Make Packet's provisioning systems less dependent on third party network providers and inter-datacenter connectivity

   View case
2. IL-0396 New or live-migrated GCE instances, VPN tunnels and network load balancers were unreachable on public IPs for over 2h

   Google Cloud2017

   Symptoms

   On 30 January 2017, Google Compute Engine instances, Cloud VPN tunnels and GCE network load balancers that were created or live migrated between 10:36 and 12:42 PDT could not be reached on their public IP addresses. Resources created earlier, and instances that were not live migrated, kept working.

   Network unreachableProbe failures

   Trigger

   A large set of updates was applied to a rarely used load balancing configuration on the shared layer 2 load balancers.

   Root cause

   Software defect

   Component

   Cloud load balancerNetwork control plane (SDN)

   Contributing factor

   Excessive blast radiusGap in automated tests

   Remediation

   • Restarted the jobs that program changes into the network load balancers, so the stuck changes were processed as a batch
   • Applied the fix zone by zone between 11:36 and 12:42

   Action items

   • Increase the canary timeout so updates that hit the inefficient code path only slow network changes instead of stopping them
   • Improve the inefficient code path
   • Write new tests that check behaviour on a wider range of configurations
   • Accelerate replacing global propagation of address configuration with decentralized routing
   • Add metrics and alerting that identify this kind of issue sooner

   View case

By root cause5