Incident Library
Applied filters Root causeDependency failureOrganisationLinodeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
    Symptoms

    On 1 April 2025 users of Linode Kubernetes Engine (LKE) began having trouble connecting to their clusters' control planes, and the LKE Dashboard was affected as well. Services inside the clusters kept working. The first status update was posted at 20:28 UTC; at 21:33 UTC Linode said it was investigating an issue with DNS functionality within LKE.

    DNS resolution failuresControl plane unresponsive
    Trigger

    An earlier update to a database that the LKE DNS system depends on caused the DNS service to hang.

    Root cause
    Dependency failure
    Component
    Authoritative DNS servers and recordsApplication database
    Contributing factor
    –
    Detection

    The first status update, at 20:28 UTC on 1 April, said Linode was investigating and that users might have difficulty connecting to their cluster control plane. At 21:33 UTC Linode said the issue affected DNS functionality within LKE, and at 22:29 UTC that it had identified the underlying cause.

    Remediation
    • restarted the affected LKE DNS services
    Action items
    • Improve how the DNS services recover from transient connectivity failures to their dependencies
    • Improve monitoring to detect similar problems sooner
    • Update the deployment process to better coordinate changes between dependent systems
    Lessons

    A change to a database that the DNS service depends on can make the DNS service hang, which here cut off external access to the clusters and the LKE Dashboard while services inside the clusters kept working. The DNS service should recover by itself from transient failures of its dependencies, and changes to dependent systems need to be coordinated.

    View case
  2. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters
    Linode2025
    Symptoms

    From about 21:38 UTC on 5 February 2025, image pulls from Docker Hub over IPv6 failed with 429 Too Many Requests errors across several Linode regions. Both unauthenticated and authenticated pull requests were refused.

    Image pull failures
    Trigger

    Docker Hub's abuse prevention measures started rate-limiting image pulls coming from Linode's IPv6 address space.

    Root cause
    Dependency failure
    Component
    Container registry and its credentialsVPC DNS resolver
    Contributing factor
    Unknown limit
    Detection

    The rate limiting began at about 21:38 UTC. The first status post, at 22:48 UTC, said Linode was aware of an issue with Docker Hub over IPv6 causing rate limiting and was investigating, and at 01:06 UTC on 6 February the status page said the issue had been identified as rate limiting of image pull requests to Docker Hub over IPv6.

    Remediation
    • changed the DNS resolvers to return an A record instead of an AAAA record so image pulls to Docker Hub used IPv4
    Action items
    • Find long-term solutions so that Docker Hub's abuse prevention measures do not affect Linode services
    Lessons

    A registry's abuse limit that groups IPv6 addresses by their first 64 bits can block a provider's whole address range at once, and authentication does not help. Answering DNS lookups with IPv4 addresses only was the quick way to move image pulls off the blocked range.

    View case
Check your cluster for this

These 2 cases involve Application database and Authoritative DNS servers and records. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure2 cases

Component
Application database1Authoritative DNS servers and records1Container registry and its credentials1VPC DNS resolver1
Contributing factor
Unknown limit1
Organisation
Linode2
  1. IL-0162 LKE users could not reach their cluster control planes or the LKE Dashboard because an earlier database update made the internal LKE DNS service hang
    Linode2025
  2. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters
    Linode2025