Cases13 cases

6 to 10 of 13

1. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters

   Linode2025

   Symptoms

   From about 21:38 UTC on 5 February 2025, image pulls from Docker Hub over IPv6 failed with 429 Too Many Requests errors across several Linode regions. Both unauthenticated and authenticated pull requests were refused.

   Image pull failures

   Trigger

   Docker Hub's abuse prevention measures started rate-limiting image pulls coming from Linode's IPv6 address space.

   Root cause

   Dependency failure

   Component

   Container registry and its credentialsVPC DNS resolver

   Contributing factor

   Unknown limit

   Detection

   The rate limiting began at about 21:38 UTC. The first status post, at 22:48 UTC, said Linode was aware of an issue with Docker Hub over IPv6 causing rate limiting and was investigating, and at 01:06 UTC on 6 February the status page said the issue had been identified as rate limiting of image pull requests to Docker Hub over IPv6.

   Remediation

   • changed the DNS resolvers to return an A record instead of an AAAA record so image pulls to Docker Hub used IPv4

   Action items

   • Find long-term solutions so that Docker Hub's abuse prevention measures do not affect Linode services

   Lessons

   A registry's abuse limit that groups IPv6 addresses by their first 64 bits can block a provider's whole address range at once, and authentication does not help. Answering DNS lookups with IPv4 addresses only was the quick way to move image pulls off the blocked range.

   View case
2. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device

   UK Ministry of Justice2023

   Symptoms

   At 13:42 the RootVolUtilisation-Critical alert fired in the high-priority alerts channel. At 14:03 a user reported image pull errors with the message "no space left on device".

   Node disk fullImage pull failuresPod evictionPods stuck in ContainerCreatingIP address exhaustion

   Trigger

   Upgrading the EKS Terraform module to version 18, which changed the node launch template, and draining workloads onto the new nodes it created.

   Root cause

   Configuration change

   Component

   Node root diskNode launch templateNode poolAWS VPC CNICoreDNS

   Contributing factor

   Missed in reviewNo configuration testsNo alert on limit

   Detection

   The RootVolUtilisation-Critical alert fired in the high-priority alerts channel at 13:42, and a user reported "no space left on device" image pull errors at 14:03.

   Remediation

   • Uncordoned the old default node group and drained the new nodes to move workloads back
   • Identified the launch template change that cut the root volume to 20 GB and applied a fix on the manager and live clusters
   • Drained the old node group onto the fixed nodes

   Action items

   • Update the runbook to compare launch template changes during an EKS module upgrade
   • Create a test setup that pulls images of different sizes, like live
   • Update the RootVolUtilisation alert runbook to check the disk space configuration
   • Scale CoreDNS dynamically based on the number of nodes
   • Investigate whether IPv6 can solve the IP prefix starvation problem
   • Add drift testing to identify when a terraform plan shows a change to the launch template
   • Set up logging for the CNI and ipamd logs and alerts on errors related to IP prefix starvation

   Lessons

   The write-up lists review actions rather than lessons. They aim at two gaps: the launch template changes brought by a module upgrade were not compared or drift-tested before rollout, and there were no CNI or ipamd logs or alerts to show the IP prefix starvation the team suspected during the drain.

   View case
3. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images

   UK Ministry of Justice2023

   Symptoms

   At 14:22 a Cloud Platform team member noticed failed Docker Hub logins. Fetching an image returned: failed to fetch manifest: Head "https://registry-1.docker.io/v2/ministryofjustice/cloud-platform-tools/manifests/2.1": toomanyrequests: too many failed login attempts for username or IP address

   Image pull failures

   Trigger

   The Docker Hub password was changed as part of LastPass remediation activities while the EKS cluster nodes and Concourse still logged in with the previous password.

   Root cause

   Operator error

   Component

   Container registry and its credentialsNode launch templateNode poolCI system

   Contributing factor

   Credential in configurationShared credential

   Detection

   A Cloud Platform team member saw the failed Docker Hub login error at 14:22. The Kuberhealthy daemonset check showed at 14:52 that the cluster could not pull images, and a user reported ErrImagePull in #ask-cloud-platform at 14:54.

   Remediation

   • Changed the Docker Hub password back to the previous value still used by the EKS cluster nodes and Concourse
   • Confirmed with the reporting user that the pod deployed and that cluster image pulls worked again

   Action items

   • Switch from the Docker Hub username and password to a Docker Hub token created specifically for Cloud Platform
   • Rotate the password cleanly out of hours with a graceful recycle of the nodes

   Lessons

   The write-up states no separate lessons. Its review action replaces the Docker Hub username and password with a Docker Hub token created specifically for Cloud Platform.

   View case
4. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff

   Microsoft Azure2022

   Symptoms

   From 06:00 UTC on 30 August 2022, Azure VMs running Ubuntu 18.04 (bionic) with unattended-upgrades enabled began failing DNS resolution and losing network connectivity. Several Azure teams, among them the AKS and Azure Container Apps teams, were alerted by production alerts shortly after the update packages were published.

   DNS resolution failuresImage pull failuresMonitoring outage

   Trigger

   Unattended-upgrades on Ubuntu 18.04 VMs installed a Canonical security update published at 06:00 UTC on 30 August 2022, including systemd version 237-3ubuntu10.54.

   Root cause

   Breaking change in an upgrade

   Component

   Node operating system imagekubeletContainer registry and its credentialsClient-side DNS resolver and cache

   Contributing factor

   No staged rolloutExcessive blast radiusAutomation amplification

   Detection

   Several Azure teams, including AKS and Azure Container Apps, were alerted by production alerts shortly after the packages were published; incoming support cases then confirmed which Ubuntu versions were affected.

   Remediation

   • pointed customers to Canonical's description of the systemd bug and its fix
   • recommended rebooting affected VMs or running a template script that repairs DNS resolution
   • built automatic detection and remediation for AKS nodes and rolled it out across all regions
   • mitigated the remaining AKS customers manually through support

   Action items

   • Review the process that lets critical security updates bypass Safe Deployment Practices
   • Improve monitoring of the AKS data plane to alert on an uptick of errors
   • Review AKS monitoring algorithms so that similar VM errors are detected and alerted on
   • Establish an improved escalation path to Canonical during outages and add it to internal Technical Service Guides
   • AKS takes full control of the node security patch mechanism, with additional testing and controlled release of patches (estimated December 2022)
   • Work with Canonical to run dedicated tests on proposed packages before they are published for Azure IaaS VMs
   • Establish a process with Canonical to close the testing gap for the upgrade scenario (estimated March 2023)
   • Give AKS customers maintenance window control and environment staging for these patches (estimated December 2023)

   Lessons

   Security patches applied automatically by unattended-upgrades, once a day by default, and outside Azure's Safe Deployment Practices reached Ubuntu 18.04 VMs in every Azure region, so one systemd regression became a DNS outage across Azure. On AKS, lost node DNS showed up as image pull failures such as ImagePullBackoff, because kubelet could not resolve the registry. Azure plans to put node security patching under AKS's control with testing and controlled release; customers can use automatic node image upgrade for AKS, or automatic VM guest patching for VMs, instead of unattended-upgrades.

   View case
5. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters

   Gitpod2022

   Symptoms

   At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

   Image pull failuresDNS resolution failuresService outageData loss

   Root cause

   DNS or network fault

   Component

   CoreDNSContainer registry and its credentialsObject storage

   Contributing factor

   Single point of failure

   Detection

   The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

   Remediation

   • Created new workspace clusters in the EU and the US with a werft job
   • Redirected traffic to the new clusters

   Action items

   • Keep workspace data on cloud provider persistent volumes that survive node and cluster failures
   • Let users download their files from a workspace that cannot be started
   • Improve the workspace backup process
   • Use three DNS servers instead of one: a GCP-internal one, 8.8.8.8 and Cloudflare's 1.1.1.1
   • Do more DNS resiliency work, tracked in an epic
   • Improve support for multi-region clusters and later go multi-cloud

   View case

By root cause8