Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases33 cases

33 to 33 of 33

  1. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017
    Symptoms

    While running the CI process on a local machine to test an application Helm chart, the SRE lead watched the pods with watch kubectl get pods. All pods reached Running, and then some went back to Pending. Two hours later the problem had grown and some pods showed the phase Unknown.

    Pods PendingImage pull failuresNode NotReadyService outage
    Trigger

    A CI run installed the application's Helm charts for all four markets (Efritin.com, Tonaton.com, Ikman.lk and Bikroy.com) on the nonproduction cluster at the same time.

    Root cause
    Capacity exhaustion
    Component
    Node root diskkubeletContainer runtimeHelm
    Contributing factor
    Unreviewed defaultsMonitoring blind spotIgnored early signalSingle point of failure
    Detection

    The SRE lead saw pods go from Running back to Pending while watching kubectl get pods during the CI run; no monitoring raised an alert, even when all three workers were not ready.

    Remediation
    • terminated the three worker nodes and let the auto scaling group replace them
    Action items
    • Fix the Datadog integration for the nonproduction AWS account and repeat the installs with telemetry
    • Increase the worker root EBS volumes from 20GB to 120GB
    • Keep common base images pulled on every node with a DaemonSet until kops supports user hooks
    • Configure different allocation limits for the different layers in Kubernetes
    • Discuss a highly available Tiller with the Helm team
    • Find a way to throttle the number of newly created pods
    Lessons

    An ordinary deploy of the application could collapse the cluster, because the default 20GB gp2 root volumes could not keep up with parallel image pulls and container starts. Without node metrics and pod phase metrics, three NotReady workers raised no alert. A single Tiller replica can block the deploy that would fix an outage.

    View case
Check your cluster for this

These 33 cases involve Kubernetes API server and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause11

Dependency failure3 cases

Component
Cloud block storage1Container runtime1Credential broker for pods1Database1Data center power and cooling1etcd1InfiniBand fabric1Persistent volumes and stateful sets1Service discovery1Telemetry agent1
Contributing factor
Single point of failure3Monitoring blind spot2Automation amplification1Excessive blast radius1Unpatched known bug1Harmful manual action1No alert on limit1No runbook1Overlapping failures1
Organisation
Magic Pages1Nebius1Target1
  1. IL-0133 A storm shut down cooling at Nebius's us-central1 data center, and a disk hot-plug bug kept Managed Kubernetes control planes down for hours after power returned
    Nebius2026
  2. IL-0489 Rejoining a failed node of Magic Pages' three-node MySQL cluster broke its metadata and took customer websites down
    Magic Pages2025
  3. IL-0073 Intermittent Kafka connectivity overloaded Docker daemons, set off a node rescheduling cascade and poisoned the Consul gossip mesh
    Target2019

Cloud provider outage3 cases

Component
AKS control plane1CDN and edge network1Cloud load balancer1Cluster autoscaler1Database1Ingress controller1Node pool1Web application firewall rules1
Contributing factor
Single point of failure3Excessive blast radius1Pages not delivered1
Organisation
2i2c1ANDPAD1Microsoft Azure1
  1. IL-0438 Builder Work became almost entirely unreachable with 504 errors when the Amazon CloudFront VPC Origins feature failed
    ANDPAD2026
  2. IL-0145 A power sag and chiller failure in one Australia East availability zone took down the AKS control plane's compute and its non-zone-redundant SQL Database operation queue, blocking AKS create, update and delete operations in Australia East and Australia Southeast
    Microsoft Azure2023
  3. IL-0096 A GKE node pool pinned to the europe-west2-b zone could not scale for 6.5 hours because Google ran out of resources in that zone (ZONE_RESOURCE_POOL_EXHAUSTED)
    2i2c2022

Capacity exhaustion13 cases

Component
kubelet4Database3Ingress controller3Pod scheduling and placement3Kubernetes API server2Container memory limit2Node pool2Container registry and its credentials2Admission webhook1In-memory cache1CI system1Cloud controller manager1conntrack table1Container runtime1CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Helm1Istio1kube-dns1Linux kernel1Managed Kubernetes control plane and provider API1NAT gateway1Node root disk1Persistent volumes and stateful sets1
Contributing factor
Single point of failure13Unreviewed defaults4Monitoring blind spot3Retry storm3Automation amplification2Excessive blast radius2Unknown limit2Harmful manual action2No alert on limit2No caching2Unverified assumption1Ignored early signal1Limited debug access1No runbook1No headroom1No staged rollout1Ineffective rollback1Unrepresentative staging1
Organisation
Backpackr2Zalando2Datadog1IONOS Cloud1Magic Pages1Mercari1Moonlight1PostHog1Saltside1Scaleway1UK Ministry of Justice1
  1. IL-0465 IONOS Container Registry image pushes and pulls failed with 504 Gateway Timeout
    IONOS Cloud2026
  2. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  3. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  4. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  5. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  6. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  7. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  8. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  9. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  10. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  11. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  12. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  13. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Configuration change1 case

Component
In-memory cacheContainer memory limitPersistent volumes and stateful sets
Contributing factor
Unverified assumptionNo health probesNo headroomRetry stormSingle point of failure
Organisation
AdGuard VPN
  1. IL-0063 VPN outage after recreating a Redis StatefulSet restarted the whole Redis cluster without readiness checks
    AdGuard VPN2025

Breaking change in an upgrade2 cases

Component
Admission webhook1Calico1ConfigMaps and Secrets1Client-side DNS resolver and cache1etcd1GKE control plane1Kubernetes API server1Node pool1Persistent volumes and stateful sets1
Contributing factor
Single point of failure2Untested backup1Configuration not in code1Monitoring blind spot1No headroom1Ineffective rollback1Deferred upgrades1Unrepresentative staging1
Organisation
Reddit1Wealthy1
  1. IL-0101 After Google forced a GKE control plane upgrade to 1.33.2 at a deprecation deadline, no new node could initialise, so autoscaling and spot node replacement stopped for 15 hours
    Wealthy2025
  2. IL-0060 Kubernetes 1.24 upgrade removed the master node label that Calico route reflectors selected on, dropping all pod routes and taking Reddit down for 314 minutes
    Reddit2023

Software defect3 cases

Component
Kubernetes API server2Database1etcd1GPU driver1kubelet1Linkerd1Linux kernel1Container memory limit1Persistent volumes and stateful sets1
Contributing factor
Single point of failure3Monitoring blind spot2Unverified assumption1Change with a delayed effect1Ignored early signal1Harmful manual action1No staged rollout1Overlapping failures1Ineffective rollback1
Organisation
Barush Mendez1Monzo1Palark1
  1. IL-0138 A failed Nvidia driver build left broken kernel packages on the only Kubernetes master node, which hit a kernel panic on reboot after a power outage and took the homelab cluster down for six days
    Barush Mendez2025
  2. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data
    Palark2020
  3. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Hardware failure1 case

Component
ConfigMaps and SecretsetcdKubesprayNode root disk
Contributing factor
Single point of failure
Organisation
Hans H. Kim
  1. IL-0185 A disk failure made the first control plane node of a Kubespray cluster read-only, and recovery needed the Kubespray inventory reordered and a new control plane node joined
    Hans H. Kim2023

Certificate or credential expiry2 cases

Component
cert-manager1GKE control plane1Kubernetes API server1Node pool1RBAC and service accounts1Secrets store1
Contributing factor
Single point of failure2Unverified assumption1Limited debug access1Harmful manual action1Monitoring blind spot1
Organisation
GoCardless1Webhook Relay1
  1. IL-0113 The managed GKE control plane of Webhook Relay's main cluster failed because its API server certificates had never been rotated, and a manual credential rotation reported success without taking effect
    Webhook Relay2022
  2. IL-0460 Vault served an expired TLS certificate and almost every GoCardless service pod crash-looped, a 22-minute full outage
    GoCardless2020

DNS or network fault2 cases

Component
Container registry and its credentials2CoreDNS1Container memory limit1Object storage1
Contributing factor
Single point of failure2Harmful manual action1Overlapping failures1
Organisation
GitHub1Gitpod1
  1. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
  2. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry
    GitHub2020

Faulty deployment2 cases

Component
Admission webhook1CI system1Ingress controller1Node pool1
Contributing factor
Single point of failure2Unverified assumption1Change started before off-hours1Change with a delayed effect1No alert on limit1
Organisation
Kubernetes project1UK Ministry of Justice1
  1. IL-0088 A validating admission webhook added by a new ingress controller module version made the live-1 cluster reject every new Ingress with an x509 error
    UK Ministry of Justice2020
  2. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018