Cases2 cases

1 to 2 of 2

1. IL-0568 A layer 2 loop after an access switch change broke network traffic across a whole DeNA site, with mass MACFLAP logs

   DeNA2026

   Symptoms

   At about 20:00 on a weekday, one of DeNA's largest sites reported that terminals of a customer-facing system could not communicate and business was suffering. The network group's monitoring had raised no alert.

   Intermittent connection failuresHigh latency

   Trigger

   During a routine change on 15 Cisco Catalyst 2960L access switches at the site, which tidied descriptions, added VLAN definitions and added allowed VLANs on trunk ports, `no switchport trunk allowed vlan` was applied to GigabitEthernet0/15 of switch SW3, only one of the two physical ports in the LACP bundle between SW3 and SW2.

   Root cause

   Operator error

   Component

   Network switch

   Contributing factor

   Monitoring blind spotNo staged rolloutNo runbookChange during peak hours

   Detection

   The site reported the problem at about 20:00; no monitoring alert fired, because the affected port had no interface monitoring and LACP monitoring had been removed by an earlier monitoring change.

   Remediation

   • Rolled back the switch changes without knowing the cause; service recovered when the interface setting of one switch was restored
   • Found the faulty port and confirmed there was no other impact, completing the first response at about 22:00

   Action items

   • Fix the missing interface and LACP monitoring
   • Require a review for every monitoring change
   • Block merges to the alert rule repository without a reviewer's approval
   • Enable automatic AI review on pull requests
   • Announce every network change in advance, however minor
   • Agree the work date with the site's managers and avoid the site's event days
   • Prepare rollback procedures for high-impact maintenance and for work on several devices at once

   Lessons

   Any change can cause an outage, through a mistake, an unexpected event or a bug, so changes need to be scheduled with a maintenance window agreed in advance. A change on a downstream access switch can affect the whole site. Changing many devices at once made it slow both to find the faulty device and to roll back, and the parts that can be prevented by mechanisms rather than by people's care, such as reviews and monitoring, should be enforced that way.

   View case
2. IL-0562 Most Linodes in Linode's Dallas data center had intermittent latency and reachability loss from broadcast storms

   Linode2022

   Symptoms

   At about 00:05 UTC, Linode's monitoring systems alerted on reachability problems in the Dallas data center. About 80-90% of the Linodes there had intermittent latency and lost reachability from time to time.

   Intermittent connection failuresHigh latencyIntermittent timeouts

   Root cause

   DNS or network fault

   Component

   Network switch

   Contributing factor

   –

   Detection

   Monitoring systems alerted on reachability problems in the Dallas data center at about 00:05 UTC, and an incident was raised at 00:10.

   Remediation

   • Deployed a filter to the fleet that stemmed the broadcasts, so hosts regained CPU and answered ARP requests
   • Tested the temporary filter further and made it permanent

   Action items

   • Find the root cause of the broadcast storms, taking the other recent Dallas incidents into account
   • Put processes in place to keep the infrastructure stable while upgrades are applied to the active network

   View case

By root cause2