IL-0539 KT · 2021-10-25networking
KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in Busan
KT's routers build their routing tables from two protocols: BGP (Border Gateway Protocol) for routes between KT's network and outside networks, and IS-IS (Intermediate System-to-Intermediate System) for routes inside KT's network. On 25 October 2021, partner company workers were replacing an enterprise network router in KT's Busan office, a grade C facility, working remotely from the Busan communication center, a grade A facility. The network operations center had approved the work for 01:00 to 06:00 on 26 October, but it was done during the day on 25 October, by partner company staff without a KT work manager present and with the network still connected. The KT manager and the partner company staff agreed together to work during the day, and the manager was away on other work.
The routing configuration script in the work plan lacked the 'exit' command that should have closed the IS-IS commands. The ministry understood that such scripts are written by the partner company and KT staff together, and KT staff checked this one in a first and a second pre-verification, both done by people reading it, without finding the omission. There was no virtual testbed to run the work on a disconnected network first. Without the 'exit', the following commands were joined to BGP, so route information meant to be exchanged by BGP was sent into IS-IS. IS-IS normally exchanges about 10,000 entries in KT's network; hundreds of thousands of BGP entries were loaded into it, and the routing paths went wrong. IS-IS routers exchange updates automatically, so the wrong routes set on the Busan router reached IS-IS routers in other regions. KT's IS-IS connects the whole country as one network with no safeguard against wrong data, and there was no system to stop a regional error from spreading nationwide, so the mistake on one router spread to routers across the country. The joint investigation team of the Ministry of Science and ICT and outside experts also ruled out a DDoS attack from IP patterns and traffic.
Recovery began at 11:57. KT first isolated the router where the problem started, then logged in to each regional router, checked it, deleted the faulty processes and brought them back up. The ministry issued a level 2 alert, of four levels, at 11:56, and KT reported recovery complete at 12:45.
- Severity
- –
- Scope
- KT's internet service nationwide for about 89 minutes; added load on KT's voice and text message networks
- Users affected
- –
- Region
- nationwide across KT's network in Korea
- Time to detect
- 4m
- Time to resolve
- 1h 29m
Symptoms
On 25 October 2021 from about 11:16, traffic to the DNS servers in KT's network rose sharply, and then KT's internet service failed across the whole country. KT recognised the internet outage at 11:20, and the outage lasted about 89 minutes until KT finished recovery at about 12:45.
The DNS server logs showed traffic more than 22 times normal on the central primary DNS, more than 4 times normal on the secondary DNS and more than 3.7 times normal on the Busan DNS. Packet analysis found at most about 15 DNS queries from any single IP address, only for normal domains such as Naver and Daum, and no repeated queries for changing, nonexistent domains. The incoming traffic reached at most 8% of the central primary DNS server's bandwidth and 28% of the Busan DNS server's bandwidth.
From around 11:20, KT first suspected a DDoS attack, and the outage was widely taken for a cyber attack. KT reported the outage to the Ministry of Science and ICT at 11:40 and at 11:44 told the ministry that it was a routing error, not a DDoS attack. The ministry, which had also seen the traffic increase in its own monitoring and sent investigators, judged from packet analysis that it was not a DDoS attack. Firewalls did not act, because they block attacks from outside and do not stop a configuration error made inside the network.
KT's IPTV, voice and text message networks are separate from its internet network, but the investigation estimated that load on the voice and text networks grew as people turned to calls and texts and as many users reset their devices. During recovery, logging in to routers in the regions was difficult because the internet itself was not working.
Root cause
Root cause Configuration change
- Trigger
- During the replacement of an enterprise network router at KT's Busan office, worked on remotely from the Busan communication center, a worker entered a routing configuration script that lacked the 'exit' command that should have ended the IS-IS part of the configuration.
Detection
KT recognised the internet outage at 11:20, four minutes after DNS traffic began to surge at 11:16. The Ministry of Science and ICT saw the traffic increase in its own monitoring, and KT reported the outage to the ministry at 11:40.
Remediation
- Isolated the router where the faulty routing configuration was entered
- Logged in to each regional router, deleted the faulty processes and restarted them
Action items
- Check the network management of major carriers, including their work procedures and the systems that stop technical errors from spreading
- Consider having major carriers introduce simulation systems that diagnose errors from network work in advance
- Have the NMS center and network operations center of each major carrier check technically that approved work plans are followed
- Strengthen safeguards in routing work, such as limiting the number of route entries updated at once below a set level
- Strengthen the monitoring of carriers' outages and develop technology, including AI-based preventive diagnosis, for network stability and resilience
- Prepare structural measures for a stable network architecture and network survivability
- Let carriers notify users of outages through SNS, text messages and other means, not only the website
Lessons
The investigation named management problems and technical problems. On the management side, work approved for the night was done during the day, partner company workers did the routing work without a work manager, and the work was done with the network still connected. On the technical side, the two pre-verification steps were people reading the script and missed the missing 'exit', there was no virtual testbed for trying the work on a disconnected network, and nothing stopped an error in one region from spreading to the whole country.
Sources
Related cases
IL-0225An IGP purge command made every Microsoft WAN router recompute routes, causing global latency, timeouts and packet lossMicrosoft Azure · 2023same root cause · shared Interior routing (IGP), BGP routing
IL-0228IPv4 routing across OVHcloud's global network broke after a router change put the full internet routing table into OSPFOVHcloud · 2021same root cause · shared Interior routing (IGP), BGP routing
IL-0400Public sector sites across Wales lost connectivity for over 8 hours as a PE router wrongly advertised a default routePSBA · 2026same root cause · shared BGP routing
IL-0394A Cloudflare router in Miami leaked IPv6 BGP routes to peers and providers for 25 minutes, congesting the backboneCloudflare · 2026same root cause · shared BGP routing
IL-0266Eqiad routers looped traffic between each other after a BGP MED change and could not reach other sites' public IPsWikimedia Foundation · 2021same root cause · shared BGP routing
Check your cluster for this
For AI tools and scripts
- This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0539
- The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0539/similar
- MCP server (Streamable HTTP, no key):
https://incidentlib-mcp.benchlet.com/mcp
The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.