Cases2 cases

1 to 2 of 2

1. IL-0568 A layer 2 loop after an access switch change broke network traffic across a whole DeNA site, with mass MACFLAP logs

   DeNA2026

   Symptoms

   At about 20:00 on a weekday, one of DeNA's largest sites reported that terminals of a customer-facing system could not communicate and business was suffering. The network group's monitoring had raised no alert.

   Intermittent connection failuresHigh latency

   Trigger

   During a routine change on 15 Cisco Catalyst 2960L access switches at the site, which tidied descriptions, added VLAN definitions and added allowed VLANs on trunk ports, `no switchport trunk allowed vlan` was applied to GigabitEthernet0/15 of switch SW3, only one of the two physical ports in the LACP bundle between SW3 and SW2.

   Root cause

   Operator error

   Component

   Network switch

   Contributing factor

   Monitoring blind spotNo staged rolloutNo runbookChange during peak hours

   Detection

   The site reported the problem at about 20:00; no monitoring alert fired, because the affected port had no interface monitoring and LACP monitoring had been removed by an earlier monitoring change.

   Remediation

   • Rolled back the switch changes without knowing the cause; service recovered when the interface setting of one switch was restored
   • Found the faulty port and confirmed there was no other impact, completing the first response at about 22:00

   Action items

   • Fix the missing interface and LACP monitoring
   • Require a review for every monitoring change
   • Block merges to the alert rule repository without a reviewer's approval
   • Enable automatic AI review on pull requests
   • Announce every network change in advance, however minor
   • Agree the work date with the site's managers and avoid the site's event days
   • Prepare rollback procedures for high-impact maintenance and for work on several devices at once

   Lessons

   Any change can cause an outage, through a mistake, an unexpected event or a bug, so changes need to be scheduled with a maintenance window agreed in advance. A change on a downstream access switch can affect the whole site. Changing many devices at once made it slow both to find the faulty device and to roll back, and the parts that can be prevented by mechanisms rather than by people's care, such as reviews and monitoring, should be enforced that way.

   View case
2. IL-0349 IONOS Private Cloud in Berlin lost management access, storage and VM internet routing after a VLAN change

   IONOS Cloud2026

   Symptoms

   On 15 January 2026, network services for VMware-based Private Clouds in IONOS Cloud's Berlin location stopped working: customers lost management VPN access and traffic to and from their virtual machines. Inside IONOS, several management interfaces in the Berlin region could no longer be reached.

   Network unreachableStorage disconnected or stale mounts

   Trigger

   During a planned maintenance window to add a new VLAN to the management devices, a human error in the command replaced the existing VLANs on the trunk between the core and management devices instead of adding to them, so only the new VLAN stayed active.

   Root cause

   Operator error

   Component

   Network switchTransit gateway or network interconnect

   Contributing factor

   Excessive blast radiusNo safeguard on destructive change

   Remediation

   • Restarted the affected storage services in the final stage of recovery, which caused a brief second outage

   Action items

   • Audit the management device architecture to segment it better (target Q3 2026)
   • Evaluate and implement ways to prevent unintended VLAN changes, such as improved configuration safeguards or automation scripts (target Q1 2026)

   View case

By root cause1