Cases1 case

1 to 1 of 1

1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang

   Linode2025

   Symptoms

   On 1 April 2025 users of Linode Kubernetes Engine (LKE) began having trouble connecting to their clusters' control planes, and the LKE Dashboard was affected as well. Services inside the clusters kept working. The first status update was posted at 20:28 UTC; at 21:33 UTC Linode said it was investigating an issue with DNS functionality within LKE.

   DNS resolution failuresControl plane unresponsive

   Trigger

   An earlier update to a database that the LKE DNS system depends on caused the DNS service to hang.

   Root cause

   Dependency failure

   Component

   Authoritative DNS servers and recordsApplication database

   Contributing factor

   –

   Detection

   The first status update, at 20:28 UTC on 1 April, said Linode was investigating and that users might have difficulty connecting to their cluster control plane. At 21:33 UTC Linode said the issue affected DNS functionality within LKE, and at 22:29 UTC that it had identified the underlying cause.

   Remediation

   • restarted the affected LKE DNS services

   Action items

   • Improve how the DNS services recover from transient connectivity failures to their dependencies
   • Improve monitoring to detect similar problems sooner
   • Update the deployment process to better coordinate changes between dependent systems

   Lessons

   A change to a database that the DNS service depends on can make the DNS service hang, which here cut off external access to the clusters and the LKE Dashboard while services inside the clusters kept working. The DNS service should recover by itself from transient failures of its dependencies, and changes to dependent systems need to be coordinated.

   View case

By root cause1