Cases3 cases

1 to 3 of 3

1. IL-0568 A layer 2 loop after an access switch change broke network traffic across a whole DeNA site, with mass MACFLAP logs

   DeNA2026

   Symptoms

   At about 20:00 on a weekday, one of DeNA's largest sites reported that terminals of a customer-facing system could not communicate and business was suffering. The network group's monitoring had raised no alert.

   Intermittent connection failuresHigh latency

   Trigger

   During a routine change on 15 Cisco Catalyst 2960L access switches at the site, which tidied descriptions, added VLAN definitions and added allowed VLANs on trunk ports, `no switchport trunk allowed vlan` was applied to GigabitEthernet0/15 of switch SW3, only one of the two physical ports in the LACP bundle between SW3 and SW2.

   Root cause

   Operator error

   Component

   Network switch

   Contributing factor

   Monitoring blind spotNo staged rolloutNo runbookChange during peak hours

   Detection

   The site reported the problem at about 20:00; no monitoring alert fired, because the affected port had no interface monitoring and LACP monitoring had been removed by an earlier monitoring change.

   Remediation

   • Rolled back the switch changes without knowing the cause; service recovered when the interface setting of one switch was restored
   • Found the faulty port and confirmed there was no other impact, completing the first response at about 22:00

   Action items

   • Fix the missing interface and LACP monitoring
   • Require a review for every monitoring change
   • Block merges to the alert rule repository without a reviewer's approval
   • Enable automatic AI review on pull requests
   • Announce every network change in advance, however minor
   • Agree the work date with the site's managers and avoid the site's event days
   • Prepare rollback procedures for high-impact maintenance and for work on several devices at once

   Lessons

   Any change can cause an outage, through a mistake, an unexpected event or a bug, so changes need to be scheduled with a maintenance window agreed in advance. A change on a downstream access switch can affect the whole site. Changing many devices at once made it slow both to find the faulty device and to roll back, and the parts that can be prevented by mechanisms rather than by people's care, such as reviews and monitoring, should be enforced that way.

   View case
2. IL-0562 Most Linodes in Linode's Dallas data center had intermittent latency and reachability loss from broadcast storms

   Linode2022

   Symptoms

   At about 00:05 UTC, Linode's monitoring systems alerted on reachability problems in the Dallas data center. About 80-90% of the Linodes there had intermittent latency and lost reachability from time to time.

   Intermittent connection failuresHigh latencyIntermittent timeouts

   Root cause

   DNS or network fault

   Component

   Network switch

   Contributing factor

   –

   Detection

   Monitoring systems alerted on reachability problems in the Dallas data center at about 00:05 UTC, and an incident was raised at 00:10.

   Remediation

   • Deployed a filter to the fleet that stemmed the broadcasts, so hosts regained CPU and answered ARP requests
   • Tested the temporary filter further and made it permanent

   Action items

   • Find the root cause of the broadcast storms, taking the other recent Dallas incidents into account
   • Put processes in place to keep the infrastructure stable while upgrades are applied to the active network

   View case
3. IL-0263 GitHub was slow and erroring all day and down for 18 minutes as links between access and aggregation switches saturated

   GitHub2012

   Symptoms

   On Friday, 30 November 2012, GitHub had sporadic bursts of slow responses and intermittent errors for the whole day and was completely unavailable for 18 minutes. One fileserver pair was offline for a large part of the day, which affected a small percentage of repositories; this was a side effect of the network problems on the high-availability clustering between the fileserver nodes.

   High latencyHigh error rateService outage

   Trigger

   On the morning of 30 November GitHub moved the A switches in its server cabinets onto the new aggregation switches, the second half of a migration whose B-switch half had been done the day before.

   Root cause

   Software defect

   Component

   Network switchNIC, cable and switch port link

   Contributing factor

   Overlapping failuresUnverified assumptionHarmful manual action

   Detection

   Engineers saw the problems during the maintenance itself: connectivity to the B switch was lost as the last A switch was connected, and afterwards performance stayed low and the links between access and aggregation switches showed as completely saturated.

   Remediation

   • Fixed the bridge-loop problem and returned the affected B switch to service
   • Removed the partial link failure detection setting from the access switches
   • Escalated to the network vendor, which raised the case to its highest severity level
   • Restarted core processes on the aggregation switches in a Saturday 0600 Pacific maintenance window so they learned MAC addresses again

   Action items

   • Deploy the vendor's software update that fixes the MAC learning issue to the switches
   • Closely monitor aggregation-to-access capacity and keep a workaround process ready if the problem returns
   • Build a duplicate of the network stack, from the routers through the access switches, as a staging environment for testing such changes
   • Add automated network monitoring that alerts sooner on similar issues
   • Run scheduled incident response exercises with scenarios that counter tunnel vision
   • Contact the network vendor's support at the first sign of trouble in the network

   Lessons

   GitHub had designed the maintenance to have no impact on customers and clearly failed, so it plans to invest in a staging copy of its network to test such changes and catch bugs like this one. During the response the team fixated for a long time on a bridge loop, which kept it from considering other causes. GitHub had a very positive experience with the vendor's support staff, so it will contact that support team at the first sign of network trouble.

   View case

By root cause3