Cases1 case

1 to 1 of 1

1. IL-0562 Most Linodes in Linode's Dallas data center had intermittent latency and reachability loss from broadcast storms

   Linode2022

   Symptoms

   At about 00:05 UTC, Linode's monitoring systems alerted on reachability problems in the Dallas data center. About 80-90% of the Linodes there had intermittent latency and lost reachability from time to time.

   Intermittent connection failuresHigh latencyIntermittent timeouts

   Root cause

   DNS or network fault

   Component

   Network switch

   Contributing factor

   –

   Detection

   Monitoring systems alerted on reachability problems in the Dallas data center at about 00:05 UTC, and an incident was raised at 00:10.

   Remediation

   • Deployed a filter to the fleet that stemmed the broadcasts, so hosts regained CPU and answered ARP requests
   • Tested the temporary filter further and made it permanent

   Action items

   • Find the root cause of the broadcast storms, taking the other recent Dallas incidents into account
   • Put processes in place to keep the infrastructure stable while upgrades are applied to the active network

   View case

By root cause1