Incident Library

API · MCP 한국어

API and MCP kubegap Corrections and proposals

Incident Library

API · MCP 한국어

Symptom

Image pull failures

이미지 풀 실패 · image-pull-failures

7 cases · 2017 to 2025

Root cause Capacity exhaustion2 Configuration change1 Dependency failure1 DNS or network fault1 Operator error1 Breaking change in an upgrade1

Component Container registry and its credentials5 kubelet2 Node launch template2 Node root disk2

Contributing factor Single point of failure3 Automation amplification2 Excessive blast radius2 Unknown limit2

Organisation UK Ministry of Justice2 Datadog1 GitHub1 Linode1 Microsoft Azure

Cases

IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clustersLinode · 2025
Root cause Dependency failure
IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on deviceUK Ministry of Justice · 2023
Root cause Configuration change
IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images

View on the graph

Admission webhook1

Container runtime1

Application database1

Client-side DNS resolver and cache1

Container memory limit1

Node operating system image1

VPC DNS resolver1

Harmful manual action2

Credential in configuration1

Unreviewed defaults1

Ignored early signal1

Monitoring blind spot1

No alert on limit1

No configuration tests1

No staged rollout1

Overlapping failures1

Missed in review1

Shared credential1

1

UK Ministry of Justice · 2023

Root cause Operator error

IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOffMicrosoft Azure · 2022

Root cause Breaking change in an upgrade

IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registryGitHub · 2020

Root cause DNS or network fault

IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instancesDatadog · 2019

Root cause Capacity exhaustion

IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker downSaltside · 2017

Root cause Capacity exhaustion