Incident Library

IL-0540 Korea Ministry of the Interior and Safety · 2023-11-17networking

Korea's local government administrative services and Government24 went down as a router port dropped most large packets

The Ministry of the Interior and Safety formed a cause analysis team of 29 people: 16 outside experts from companies including LG CNS, Naver Cloud and Soul Systems (소울시스템즈), 11 of whom had worked on the recovery from the start, and 13 staff from the communications, operations and security departments of the National Information Resources Service. The logs pointed to the network, so the team analysed the network devices and the server logs. The zone of the integrated authentication server holds about 150 physical servers running 19 web servers, 50 WAS servers and 56 DBMS instances, and its traffic passes through 2 routers and 4 L4 devices inside the zone and 8 routers in the national information network. The team collected each device's logs from the time of the outage and ran network load tests in three rounds, 8 runs in total (4, 2 and 2), changing the traffic to the integrated authentication server, bypassing the L4 devices, changing packet sizes, changing the number of users to 500 or 1,500 and changing the bandwidth through the L4 devices. Hacking, including outside attacks and planted spyware, was checked with the security authorities, and no sign of it was found.

The tests showed that a router lost large packets: 90% of packets of 1,500 bytes or more were lost, because some of the ports on the module that connects cables to the router were defective. The integrated authentication server could not receive the packets it needed from the router, delays piled up, and it could not do its work. No other fault was found. On 19 November at 07:00 the connection on the defective port of the Daejeon center router was moved to another port, which ended the delays. To confirm the cause, the team rebuilt an environment like the one at the time of the outage with the L4 device and the router first named as causes, and the router's packet loss was reproduced there. Services had resumed about 24 hours and ten-odd minutes after the outage began, and the cause was announced on 25 November.

The router was introduced in 2016 and was still within its service life, so age was not judged the cause, and the reason the part was damaged is hard to determine because a physical part leaves no record in the systems. The manufacturer is Cisco, and Daeshin Information & Communications (대신정보통신) maintains it. The router was covered by basic monitoring and daily visual checks of lights and signals, but the fault was in a particular part inside the module that basic monitoring does not see. The systems were built with redundancy, but redundancy takes over only when a device does not work at all; here only part of a module failed, the device was not in a state judged abnormal, and the redundancy did not take over. An LG CNS infrastructure architecture lead said the defective port carried traffic for all the services, which is why the impact was so wide.

Severity
–
Scope
Local government administrative computer services and Government24 stopped on 17 November until about 09:00 on 18 November; some functions linked to Gwangju center systems were slow until 19 November
Users affected
–
Region
National Information Resources Service, Daejeon and Gwangju centers
Time to detect
–
Time to resolve
–

Symptoms

On Friday 17 November 2023, the local government administrative computer services in Korea stopped working, and the Government24 service of the National Information Resources Service was down until the next morning.

The logs left from the time of the outage showed abnormal network sessions on the integrated authentication server, which one part of the briefing calls the integrated verification server, and many repeated log entries of an L4 device switching to an abnormal state. That L4 device had received an OS update the day before, so the network was judged the most likely place of the fault.

The L4 devices were suspected first. At 04:00 on 18 November the National Information Resources Service replaced the two L4 devices that were not working properly with higher-performance devices, ran function and load tests, and resumed Government24 and the services used by local governments at 09:00. On Sunday the ministry announced that the L4 devices were the likely cause. Later, rolling the L4 device back to its earlier OS did not make it work normally, and the same problem occurred without the OS change, so the OS update was found not to be the cause.

After the services resumed, they worked on the weekend, when traffic was low, but some functions were slow. Most of the slow functions were tied to systems at the Gwangju center, which led the team to examine the part of a Daejeon center router that connects to the Gwangju center, where a defective port was found.

Symptom Service outagePacket lossHigh latency

Root cause

Root cause Hardware failure

Component NIC, cable and switch port link
Contributing factor Monitoring blind spotGray failure
Trigger
–

Detection

–

Remediation

Action items

Lessons

The team said that hardware faults like this defective port also happen in the private sector, and that failures of parts inside a device are hard to catch in advance with basic monitoring and daily visual checks. Redundancy did not help because it takes over only when a whole device stops working, and a module that failed only in part did not make the device count as abnormal. The impact was wide because the defective port carried traffic for all the services.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, NIC, cable and switch port link among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.