Incident Library

IL-0568 DeNA · 2026-04-14 (published)networking

A layer 2 loop after an access switch change broke network traffic across a whole DeNA site, with mass MACFLAP logs

DeNA's network group runs the networks of more than 20 sites of DeNA and its subsidiaries. A project to manage the settings of every site's switches in one uniform configuration included small changes, such as aligning the rate of ip dhcp snooping limit rate 100 and the description format. The engineer saw the work as minor, did not announce it to the site and did not schedule it around the site's hours; the site, unlike an office, was open for business at 19:00. The engineer configured 15 Cisco Catalyst 2960L switches remotely in bulk and was fine-tuning them one by one. Minutes after the start, the command that removes the allowed VLAN list, which permits all VLANs, went onto only one of the two ports of the LACP link aggregation between the most downstream switch SW3 and SW2. LAG member ports must have matching settings, and the expected result was that the mismatched link would go down; instead a loop formed and spread beyond SW3 to the whole site.

The site reported the outage at about 20:00. Without knowing which change was at fault, the engineer started rolling back; with many devices and changes and no prepared rollback procedure this took about 30 minutes, and service recovered at about 21:00 when the interface setting of one switch was restored. By about 22:00 the cause had been found and no other impact remained. Monitoring had missed the problem because the port's interface monitoring had never been configured, LACP monitoring had been dropped by an earlier mistake in a monitoring change, warning logs and a CPU rise of about 10% stayed below alert thresholds, and the final resource check had not yet been reached.

Lab tests later found that the loop happened only with specific models. Between 2960C pairs, 2960S pairs and 2960C with 2960S the LAG suspended the port and traffic on it stopped, but with SW2 as a 2960C or 2960S and SW3 as a 2960L, frames passed through a port that showed Status: up, Protocol: down and ifOperStatus=2, matching the incident. Because the port was Protocol: down, STP treated it as down and did not manage it, so the healthy link kept forwarding and the misconfigured link was not blocked, leaving two active links. The 2960L was end of life, no bug report was found and it ran the latest version; the team concluded that the problem lay in operations rather than the device and did not replace it.

Severity
–
Scope
The network of one of DeNA's largest sites; terminals of a customer-facing system could not reach their servers or were very slow, affecting the business of the whole site
Users affected
–
Region
–
Time to detect
1h
Time to resolve
2h

Symptoms

At about 20:00 on a weekday, one of DeNA's largest sites reported that terminals of a customer-facing system could not communicate and business was suffering. The network group's monitoring had raised no alert.

As other network group members investigated and the site sent more reports, the picture filled in: problems at the site had started about 30 minutes before the report, which matched the time of a switch change; an unspecified number of terminals were affected, in some places the problem cleared quickly and in others it continued; terminals of the affected system could not connect to their servers or were very slow; the CPU use of the site's main switch had risen, though not enough for an alert; and the site's switches were logging large numbers of MACFLAP messages.

The first report said the cause might be the terminals and asked the network group to check the network. With no device alerts, the engineer doing the switch work at first did not think the work was involved. Even later the cause was puzzling: Prometheus showed the misconfigured port Gi0/15 as ifOperStatus=2 (down) during the incident, and STP was enabled, so no loop seemed possible.

Symptom Intermittent connection failuresHigh latency

Root cause

Root cause Operator error

Component Network switch
Contributing factor Monitoring blind spotNo staged rolloutNo runbookChange during peak hours
Trigger
During a routine change on 15 Cisco Catalyst 2960L access switches at the site, which tidied descriptions, added VLAN definitions and added allowed VLANs on trunk ports, `no switchport trunk allowed vlan` was applied to GigabitEthernet0/15 of switch SW3, only one of the two physical ports in the LACP bundle between SW3 and SW2.

Detection

The site reported the problem at about 20:00; no monitoring alert fired, because the affected port had no interface monitoring and LACP monitoring had been removed by an earlier monitoring change.

Remediation

Action items

Lessons

Any change can cause an outage, through a mistake, an unexpected event or a bug, so changes need to be scheduled with a maintenance window agreed in advance. A change on a downstream access switch can affect the whole site. Changing many devices at once made it slow both to find the faulty device and to roll back, and the parts that can be prevented by mechanisms rather than by people's care, such as reviews and monitoring, should be enforced that way.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Network switch among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.