Incident Library
Applied filters SymptomImage pull failuresDomainKubernetesClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases13 cases

13 to 13 of 13

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 13 cases involve Container registry and its credentials and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Faulty deployment2 cases

Component
CI system1Model inference server1Node pool1Container registry and its credentials1
Contributing factor
Excessive blast radius1No configuration tests1No staged rollout1Overlapping failures1Missed in review1
Organisation
IONOS Cloud1Lucidworks1
  1. IL-0237 A Kubernetes patch upgrade evicted every Ray Serve head and worker pod at once, and Lucidworks hosted LLMs went down
    Lucidworks2026
  2. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026

Capacity exhaustion4 cases

Component
Managed Kubernetes control plane and provider API2Container registry and its credentials2Admission webhook1Container runtime1Database1etcd1Helm1Kubernetes API server1kubelet1NAT gateway1Node root disk1Pod scheduling and placement1
Contributing factor
Single point of failure3Excessive blast radius2Automation amplification1Unreviewed defaults1Ignored early signal1Unknown limit1Harmful manual action1Monitoring blind spot1No alert on limit1No headroom1
Organisation
IONOS Cloud2Datadog1Saltside1
  1. IL-0464 IONOS Managed Kubernetes control planes went unreachable for minutes at a time as their shared etcd grew and fragmented
    IONOS Cloud2026
  2. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  3. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  4. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Data loss or corruption1 case

Component
Container registry and its credentials
Contributing factor
Cache-masked failure
Organisation
Harness
  1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry
    Harness2025

Dependency failure1 case

Component
Container registry and its credentialsVPC DNS resolver
Contributing factor
Unknown limit
Organisation
Linode
  1. IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clusters
    Linode2025

Configuration change1 case

Component
CoreDNSNode launch templateNode root diskNode poolAWS VPC CNI
Contributing factor
No alert on limitNo configuration testsMissed in review
Organisation
UK Ministry of Justice
  1. IL-0019 An EKS Terraform module upgrade shrank node root disks from 100 GB to 20 GB and image pulls failed with no space left on device
    UK Ministry of Justice2023

Operator error1 case

Component
CI systemNode launch templateNode poolContainer registry and its credentials
Contributing factor
Credential in configurationShared credential
Organisation
UK Ministry of Justice
  1. IL-0018 A Docker Hub password rotation left EKS nodes and Concourse locked out of Docker Hub and unable to pull images
    UK Ministry of Justice2023

Breaking change in an upgrade1 case

Component
Client-side DNS resolver and cachekubeletNode operating system imageContainer registry and its credentials
Contributing factor
Automation amplificationExcessive blast radiusNo staged rollout
Organisation
Microsoft Azure
  1. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022

DNS or network fault2 cases

Component
Container registry and its credentials2CoreDNS1Container memory limit1Object storage1
Contributing factor
Single point of failure2Harmful manual action1Overlapping failures1
Organisation
GitHub1Gitpod1
  1. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
  2. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020