Incident Library
  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases23 cases

13 to 17 of 23

  1. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
    Symptoms

    For about six months, applications on Omio's Google Kubernetes Engine clusters stalled at random: they got stuck, stopped answering health check requests or lost network connections, and nothing the team looked at explained why.

    CPU throttlingProbe failuresHigh error rateHigh latencyIntermittent timeoutsIntermittent connection failures
    Trigger

    Multi-threaded workloads running with CPU limits on nodes whose kernel did not yet have the fix for a known CFS quota bug that causes unnecessary throttling.

    Root cause
    Software defect
    Component
    Container CPU limitLinux kernelkubelet
    Contributing factor
    Unknown limitMonitoring blind spotUnpatched known bug
    Detection

    Months of random stalls and failed health checks led the team to read the cgroup cpu.stat throttling counters inside the pods, which showed far more throttling than the limits should allow.

    Remediation
    • removed CPU limits until the kernel with the CFS bug fix was deployed across all clusters
    Action items
    • Deploy the kernel with the CFS quota bug fix across all clusters
    • Monitor utilisation, saturation and error metrics, API latency and error rates to confirm the change behaves as expected
    Lessons

    A CPU limit is a time quota per 100 ms period, not a share of cores, so a multi-threaded container can be stopped for most of each period long before it uses the cores it was given, and a kernel bug can make that worse. Throttling is invisible in standard Kubernetes metrics and has to be read from the cgroup counters. Removing limits trades away protection against a container that takes a whole node, which is only acceptable when runtimes are tuned.

    View case
  2. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
    Symptoms

    Jetstack was upgrading the master of a development cluster that many teams use to test their apps during the working day, a regional Google Kubernetes Engine (GKE) cluster in europe-west1. The upgrade ran through Jetstack's GKE Terraform pipeline and did not finish before the 20-minute Terraform timeout, while the GKE console still showed the cluster as upgrading.

    Upgrade stallControl plane unresponsiveNode recreation loopService outage
    Trigger

    A planned master upgrade of the shared development cluster. The same upgrade had been done on another pre-production environment earlier that day, and no API downtime was expected.

    Root cause
    Configuration change
    Component
    Admission webhookGKE control planeKubernetes API serverkubelet
    Contributing factor
    Excessive blast radiusNo health probesMonitoring blind spotAutomation amplification
    Detection

    The first sign was the master upgrade missing the 20-minute Terraform timeout; re-running the pipeline returned the kube-apiserver unhealthy error. The cause was identified with the help of Google Support.

    Remediation
    • Deleted the OPA ValidatingAdmissionWebhook resource during a window of API server access
    Action items
    • Limit the OPA webhook to the namespaces where the policy applies and development teams have access
    • Enable the webhook only for Ingress and Service resources, the only resources the policy validates
    • Add a liveness probe so OPA is restarted when it stops responding, and update the OPA documentation
    • Follow the open upstream Kubernetes issue about this behaviour
    Lessons

    With alerting on API server response times, the team might have seen CREATE and UPDATE requests slow down across the board as soon as the OPA-backed webhook was first deployed. Every workload needs probes: OPA looked so simple to deploy that the team skipped the Helm chart, which adds a livenessProbe for the admission controller containers among other adjustments to the tutorial configuration. Others had hit the same problem, and an upstream issue to improve it was still open.

    View case
  3. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
    Symptoms

    Builds on Zalando's in-house Continuous Delivery Platform (CDP), which runs CI/CD on Kubernetes, timed out and failed for almost an hour on 11 June 2019. The Kubernetes builder pods could not get AWS IAM credentials.

    High error rateCredentials unavailableControl plane lag
    Trigger

    An unlucky rescaling of the CDP Kubernetes cluster left only one node available for builder pods. Builder pods were created and deleted on that node so quickly that kubelet fell behind.

    Root cause
    Capacity exhaustion
    Component
    kubeletCredential broker for podsKubernetes API serverCI system
    Contributing factor
    Unreviewed defaultsSingle point of failure
    Remediation
    • Scaled the CDP cluster up by hand so builder pods spread over more than one node
    Action items
    • Raise the kubelet QPS limit from the default 5 to 50 in the global cluster configuration
    View case
  4. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019
    Symptoms

    On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

    Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation
    Trigger

    A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

    Root cause
    Capacity exhaustion
    Component
    Cloud block storageNode root diskkubeletCI system
    Contributing factor
    Monitoring blind spotIgnored early signal
    Detection

    Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

    Remediation
    • Requested and received additional disk quota
    • Brought up 55 nodes with 500 GB disks in a new node pool
    • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55
    Action items
    • Create or fix monitoring so the oncall is paged when tide stops doing things
    • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support
    Lessons

    Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

    View case
  5. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019
    Symptoms

    On 25 February 2019 the Kubernetes API server of the managed Azure cluster running the data100 hub became unreachable. kubectl failed for every operation and requests to create new resources failed, so when the hub pod stopped, no new one was created and users saw a proxy error message.

    Control plane unresponsiveService outage
    Root cause
    Cloud provider outage
    Component
    AKS control planeKubernetes API serverkubelet
    Contributing factor
    Limited debug access
    Detection

    Infrastructure staff were notified in Slack at 11:34 and found that kubectl failed for all operations while the hub proxy was up.

    Remediation
    • Created a new cluster and requested a new NFS server in its resource group
    • Detached the data volumes from the old NFS server and moved them to the new cluster's resource group
    • Imported the ZFS pool into the new NFS server
    • Created the staging hub with helm and updated DNS
    • Created the prod hub with helm and updated DNS
    Action items
    • Monitor the API server endpoint remotely and alert when it is down
    View case
Check your cluster for this

These 23 cases involve kubelet and Kubernetes API server. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause8

Capacity exhaustion7 cases

Component
kubelet7Node root disk3CI system2Container runtime2Container memory limit2Cloud block storage1CoreDNS1Credential broker for pods1Client-side DNS resolver and cache1Helm1Kubernetes API server1Linux kernel1Node pool1Prometheus1
Contributing factor
Unreviewed defaults4Single point of failure4Monitoring blind spot3Ignored early signal2Unknown limit1No caching1No capacity planning1No resource requests or limits1Retry storm1
Organisation
Backpackr1Kubernetes project1LINE1Onkar Raghunath Shelke1Saltside1Scaleway1Zalando1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  3. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  4. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  5. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  6. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics
    Kubernetes project2019
  7. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017

Certificate or credential expiry2 cases

Component
Kubernetes API server2kubelet2Calico1etcd1
Contributing factor
Unreviewed defaults1Unknown limit1No alert on limit1
Organisation
ckck8031dbhang1
  1. IL-0183 kubeadm's one-year leaf certificates expired, so kube-apiserver could not reach etcd, restarted in a loop and left the cluster unreachable
    ckck8032026
  2. IL-0184 JupyterHub's KubeSpawner could not create containers and kubectl was refused because the API server and other internal certificates of a kubeadm cluster had expired
    dbhang2021

DNS or network fault1 case

Component
Kubernetes API serverkubelet
Contributing factor
–
Organisation
DigitalOcean
  1. IL-0154 Line cards on an SFO3 core router rebooted together and halved backbone capacity, and DOKS nodes whose guest networking did not recover stayed NotReady with degraded API servers until all affected nodes in the SFO3 DOKS fleet were recycled
    DigitalOcean2024

Configuration change3 cases

Component
kubelet3Admission webhook1GKE control plane1Kubernetes API server1Linux kernel1Container memory limit1Node operating system image1Prometheus1
Contributing factor
Excessive blast radius2Unreviewed defaults2Monitoring blind spot2Automation amplification1Change with a delayed effect1Limited debug access1No alert on limit1No health probes1No resource requests or limits1
Organisation
DigitalOcean1Jetstack1NU.nl1
  1. IL-0153 DOKS worker nodes went NotReady at 06:00 UTC on three days because a systemd timer in the Debian image ran apt upgrade and killed it mid kernel upgrade, breaking node networking
    DigitalOcean2023
  2. IL-0008 An unresponsive OPA validating webhook crash-looped the API server during a GKE master upgrade, and node auto-repair then recreated every node
    Jetstack2019
  3. IL-0035 Worker nodes went NotReady after a new ElastAlert rule without memory limits used up node memory
    NU.nl2019

Breaking change in an upgrade2 cases

Component
Client-side DNS resolver and cache2kubelet2CoreDNS1kube-proxy1Node operating system image1Container registry and its credentials1
Contributing factor
Automation amplification1Excessive blast radius1Harmful manual action1Monitoring blind spot1No staged rollout1Partial rollout1Ineffective rollback1
Organisation
CircleCI1Microsoft Azure1
  1. IL-0230 kube-proxy sync failures during a Kubernetes upgrade corrupted node iptables, and CircleCI jobs stopped starting
    CircleCI2023
  2. IL-0144 A systemd security update installed by unattended-upgrades lost the DNS configuration of Ubuntu 18.04 VMs, so AKS kubelets could not resolve registry names and pods failed with ImagePullBackOff
    Microsoft Azure2022

Software defect6 cases

Component
kubelet6Linux kernel2Calico1Container CPU limit1etcd1Kubernetes API server1Kubernetes controller manager1Node launch template1Linkerd1Node operating system image1Pod termination1Terraform1AWS VPC CNI1
Contributing factor
Monitoring blind spot3Unpatched known bug2No health probes2Overlapping failures2Automation amplification1Excessive blast radius1Unknown limit1Harmful manual action1No staged rollout1Ineffective rollback1Single point of failure1
Organisation
AirMap1DFDS1Google Cloud1Monzo1Oath1Omio1
  1. IL-0112 The daily node rollover put DFDS's EKS nodes on an AMI later recalled for a kernel issue, causing "PLEG is not healthy" NodeNotReady flaps and pod network timeouts
    DFDS2022
  2. IL-0139 Pods on GKE clusters with Calico network policy got stuck Terminating or Pending after upgrading to 1.22 or later because of a race condition in the Calico component
    Google Cloud2022
  3. IL-0012 CPU limits and an unpatched kernel CFS quota bug throttled multi-threaded services into stalls, failed health checks and timeouts
    Omio2020
  4. IL-0074 All AirMap platform services down as worker nodes went NotReady and pods crash-looped on network and DNS failures
    AirMap2018
  5. IL-0038 kubelet rollout hit a Kubernetes bug and left every node in a cluster NotReady, triggering mass pod eviction
    Oath2018
  6. IL-0003 linkerd lost Kubernetes service discovery after an etcd reconfiguration, and restarting every linkerd instance turned failed payments into a full platform outage
    Monzo2017

Cloud provider outage1 case

Component
AKS control planeKubernetes API serverkubelet
Contributing factor
Limited debug access
Organisation
UC Berkeley
  1. IL-0469 The managed Azure Kubernetes API server for the data100 hub stopped answering, so the stopped hub pod was not recreated
    UC Berkeley2019

unknown1 case

Component
etcdKubernetes API serverkubelet
Contributing factor
Unreviewed defaultsIgnored early signalHarmful manual actionMonitoring blind spotSingle point of failure
Organisation
NRE Labs
  1. IL-0059 Single-master Kubernetes control plane kept dying because the kubelet restarted a healthy etcd on a failing kubeadm liveness probe
    NRE Labs2018