Incident Library
Applied filters Root causeDependency failureComponentDatabaseDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
    Symptoms

    On 1 April 2025 users of Linode Kubernetes Engine (LKE) began having trouble connecting to their clusters' control planes, and the LKE Dashboard was affected as well. Services inside the clusters kept working. The first status update was posted at 20:28 UTC; at 21:33 UTC Linode said it was investigating an issue with DNS functionality within LKE.

    DNS resolution failuresControl plane unresponsive
    Trigger

    An earlier update to a database that the LKE DNS system depends on caused the DNS service to hang.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and recordsDatabase
    Contributing factor
    –
    Detection

    The first status update, at 20:28 UTC on 1 April, said Linode was investigating and that users might have difficulty connecting to their cluster control plane. At 21:33 UTC Linode said the issue affected DNS functionality within LKE, and at 22:29 UTC that it had identified the underlying cause.

    Remediation
    • restarted the affected LKE DNS services
    Action items
    • Improve how the DNS services recover from transient connectivity failures to their dependencies
    • Improve monitoring to detect similar problems sooner
    • Update the deployment process to better coordinate changes between dependent systems
    Lessons

    A change to a database that the DNS service depends on can make the DNS service hang, which here cut off external access to the clusters and the LKE Dashboard while services inside the clusters kept working. The DNS service should recover by itself from transient failures of its dependencies, and changes to dependent systems need to be coordinated.

    View case
  2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
    Symptoms

    On 15 January 2025, from about 09:55 to 11:36 CET, a database outage at Magic Pages took down customer websites. It began when the founder tried to reconnect a failed third node to the high-availability MySQL cluster and the database server reported ambiguous errors about missing metadata information.

    Service outage
    Trigger

    One of the three nodes of the MySQL cluster went down, and reconnecting it failed with errors about missing metadata information.

    Root cause
    Dependency failure
    Component
    DatabasePersistent volumes and stateful sets
    Contributing factor
    Harmful manual actionSingle point of failureMonitoring blind spot
    Remediation
    • Restarted the storage server to fix PVC provisioning
    • Let the MySQL nodes restore and sync with each other for about 45 minutes to rebuild the cluster metadata
    • Cleaned up the corrupted cluster metadata
    • Redeployed MySQL Router
    Action items
    • Monitor the health of the storage system more closely
    • Improve the failover procedures for MySQL Router
    • Investigate and fix the status page so it shows reliable information
    Lessons

    The cluster survived the first node failure as designed and lost no data, but its metadata turned out to be a single point of failure. Better monitoring of PVC health could help prevent similar issues, the 45-minute recovery leaves room for improvement, and the status page has to detect failing customer sites reliably.

    View case
Check your cluster for this

These 2 cases involve Database and Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure2 cases

Component
Database2Authoritative DNS servers and records1Persistent volumes and stateful sets1
Contributing factor
Harmful manual action1Monitoring blind spot1Single point of failure1
Organisation
Linode1Magic Pages1
  1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
  2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025