Cases2 cases

1 to 2 of 2

1. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters

   Linode2025

   Symptoms

   From about 21:38 UTC on 5 February 2025, image pulls from Docker Hub over IPv6 failed with 429 Too Many Requests errors across several Linode regions. Both unauthenticated and authenticated pull requests were refused.

   Image pull failures

   Trigger

   Docker Hub's abuse prevention measures started rate-limiting image pulls coming from Linode's IPv6 address space.

   Root cause

   Dependency failure

   Component

   Container registry and its credentialsVPC DNS resolver

   Contributing factor

   Unknown limit

   Detection

   The rate limiting began at about 21:38 UTC. The first status post, at 22:48 UTC, said Linode was aware of an issue with Docker Hub over IPv6 causing rate limiting and was investigating, and at 01:06 UTC on 6 February the status page said the issue had been identified as rate limiting of image pull requests to Docker Hub over IPv6.

   Remediation

   • changed the DNS resolvers to return an A record instead of an AAAA record so image pulls to Docker Hub used IPv4

   Action items

   • Find long-term solutions so that Docker Hub's abuse prevention measures do not affect Linode services

   Lessons

   A registry's abuse limit that groups IPv6 addresses by their first 64 bits can block a provider's whole address range at once, and authentication does not help. Answering DNS lookups with IPv4 addresses only was the quick way to move image pulls off the blocked range.

   View case
2. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances

   Datadog2019

   Symptoms

   Users reported that their jobs were not starting, with some sign that image pulls were involved. Many pods across the cluster were in error states related to image pulls, while applications that were already running were not affected.

   Image pull failuresService outage

   Trigger

   A change to the permissions on a storage bucket that one application needed to read. The application could no longer start, and its DaemonSet, running on roughly a thousand nodes of the cluster, went into a crash loop with imagePullPolicy: Always, so every restart pulled the image again although it was already on the node.

   Root cause

   Capacity exhaustion

   Component

   Container registry and its credentialsNAT gatewayApplication databaseAdmission webhook

   Contributing factor

   Automation amplificationSingle point of failureUnknown limitHarmful manual actionExcessive blast radius

   Detection

   Users reported that their jobs were not starting; monitoring had not caught it first.

   Remediation

   • replaced the affected NAT instances so registry traffic came from new source IPs
   • reconfigured the static IPs and firewall rules so applications could reach the Cloud SQL database again

   Action items

   • Add an admission webhook that rejects floating image tags, starting with latest
   • Apply the image tag webhook to workload objects only, not to pods, after it blocked pods created by in-cluster controllers

   Lessons

   The speakers' takeaway is to be very careful with DaemonSets in large clusters: they say they had broken everything DaemonSets depend on, including DNS, Vault and the image repository, and that it is very easy to break things when managing them by the thousand. imagePullPolicy: Always made every crash-loop restart pull the image again. Sending all registry traffic through three NAT instances put the whole region behind three source IPs, which the registry rate-limited, and changing those IPs cut the connections to a critical database reached over the internet through them. An admission webhook on pods rejects the pods that controllers in the cluster create, where no user sees the error.

   View case

By root cause2