Incident Library
Applied filters ComponentContainer registry and its credentialsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
    Symptoms

    Users reported that their jobs were not starting, with some sign that image pulls were involved. Many pods across the cluster were in error states related to image pulls, while applications that were already running were not affected.

    Image pull failuresService outage
    Trigger

    A change to the permissions on a storage bucket that one application needed to read. The application could no longer start, and its DaemonSet, running on roughly a thousand nodes of the cluster, went into a crash loop with imagePullPolicy: Always, so every restart pulled the image again although it was already on the node.

    Root cause
    Capacity exhaustion
    Component
    Container registry and its credentialsNAT gatewayApplication databaseAdmission webhook
    Contributing factor
    Automation amplificationSingle point of failureUnknown limitHarmful manual actionExcessive blast radius
    Detection

    Users reported that their jobs were not starting; monitoring had not caught it first.

    Remediation
    • replaced the affected NAT instances so registry traffic came from new source IPs
    • reconfigured the static IPs and firewall rules so applications could reach the Cloud SQL database again
    Action items
    • Add an admission webhook that rejects floating image tags, starting with latest
    • Apply the image tag webhook to workload objects only, not to pods, after it blocked pods created by in-cluster controllers
    Lessons

    The speakers' takeaway is to be very careful with DaemonSets in large clusters: they say they had broken everything DaemonSets depend on, including DNS, Vault and the image repository, and that it is very easy to break things when managing them by the thousand. imagePullPolicy: Always made every crash-loop restart pull the image again. Sending all registry traffic through three NAT instances put the whole region behind three source IPs, which the registry rate-limited, and changing those IPs cut the connections to a critical database reached over the internet through them. An admission webhook on pods rejects the pods that controllers in the cluster create, where no user sees the error.

    View case
Check your cluster for this

These 6 cases involve Container registry and its credentials and Admission webhook. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause6

Dependency failure1 case

Component
Container registry and its credentialsVPC DNS resolver
Contributing factor
Unknown limit
Organisation
Linode
  1. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters
    Linode2025

Configuration change1 case

Component
Kubernetes API serverRBAC and service accountsContainer registry and its credentials
Contributing factor
Monitoring blind spot
Organisation
Aqua Security
  1. IL-0068 Anonymous access to the Kubernetes API server used to plant a ClusterRoleBinding backdoor and a cryptominer DaemonSet
    Aqua Security2023

Operator error1 case

Component
CI systemNode launch templateNode poolContainer registry and its credentials
Contributing factor
Credential in configurationShared credential
Organisation
UK Ministry of Justice
  1. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023

Breaking change in an upgrade1 case

Component
Client-side DNS resolver and cachekubeletNode operating system imageContainer registry and its credentials
Contributing factor
Automation amplificationExcessive blast radiusNo staged rollout
Organisation
Microsoft Azure
  1. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022

DNS or network fault1 case

Component
Container memory limitContainer registry and its credentials
Contributing factor
Harmful manual actionOverlapping failuresSingle point of failure
Organisation
GitHub
  1. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020

Capacity exhaustion1 case

Component
Admission webhookApplication databaseNAT gatewayContainer registry and its credentials
Contributing factor
Automation amplificationExcessive blast radiusUnknown limitHarmful manual actionSingle point of failure
Organisation
Datadog
  1. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019