Cases1 case

1 to 1 of 1

1. IL-0539 KT's internet service failed nationwide for about 89 minutes as wrong routes spread from one router in Busan

   KT2021

   Symptoms

   On 25 October 2021 from about 11:16, traffic to the DNS servers in KT's network rose sharply, and then KT's internet service failed across the whole country. KT recognised the internet outage at 11:20, and the outage lasted about 89 minutes until KT finished recovery at about 12:45.

   Network unreachableService outage

   Trigger

   During the replacement of an enterprise network router at KT's Busan office, worked on remotely from the Busan communication center, a worker entered a routing configuration script that lacked the 'exit' command that should have ended the IS-IS part of the configuration.

   Root cause

   Configuration change

   Component

   Interior routing (IGP)BGP routing

   Contributing factor

   Missed in reviewNo configuration testsChange during peak hoursExcessive blast radius

   Detection

   KT recognised the internet outage at 11:20, four minutes after DNS traffic began to surge at 11:16. The Ministry of Science and ICT saw the traffic increase in its own monitoring, and KT reported the outage to the ministry at 11:40.

   Remediation

   • Isolated the router where the faulty routing configuration was entered
   • Logged in to each regional router, deleted the faulty processes and restarted them

   Action items

   • Check the network management of major carriers, including their work procedures and the systems that stop technical errors from spreading
   • Consider having major carriers introduce simulation systems that diagnose errors from network work in advance
   • Have the NMS center and network operations center of each major carrier check technically that approved work plans are followed
   • Strengthen safeguards in routing work, such as limiting the number of route entries updated at once below a set level
   • Strengthen the monitoring of carriers' outages and develop technology, including AI-based preventive diagnosis, for network stability and resilience
   • Prepare structural measures for a stable network architecture and network survivability
   • Let carriers notify users of outages through SNS, text messages and other means, not only the website

   Lessons

   The investigation named management problems and technical problems. On the management side, work approved for the night was done during the day, partner company workers did the routing work without a work manager, and the work was done with the network still connected. On the technical side, the two pre-verification steps were people reading the script and missed the missing 'exit', there was no virtual testbed for trying the work on a disconnected network, and nothing stopped an error in one region from spreading to the whole country.

   View case

By root cause1