Incident Library
Applied filters Root causeCapacity exhaustionClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases51 cases

29 to 33 of 51

  1. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
    Symptoms

    Request latency from the Temporal Server History service, the service that starts workflow executions and records their progress, was higher and less steady than it should be on a self-hosted Kubernetes deployment. The CPU dashboard showed the history pods using less than 60 percent of their CPU, yet the same pods were being throttled.

    CPU throttlingHigh latency
    Trigger

    Normal load on History pods that had a CPU limit of 2 cores on 8-core nodes, with the Go runtime's GOMAXPROCS left at its default of one per node core.

    Root cause
    Capacity exhaustion
    Component
    Container CPU limitLanguage runtime settingsPrometheus
    Contributing factor
    Unreviewed defaultsMonitoring blind spot
    Detection

    A Prometheus query for CPU throttling showed the history pods throttled although their CPU usage read under 60 percent.

    Remediation
    • set the GOMAXPROCS environment variable to 2 to match the 2-core CPU limit
    Action items
    • Set GOMAXPROCS automatically from the Kubernetes CPU limit, from Temporal release 1.21.0
    Lessons

    A CPU usage graph averaged over 15 seconds or more can sit far below 100 percent while a container is being throttled, so throttling needs its own metric. A Go program in a container should have GOMAXPROCS matched to its CPU limit, or to its CPU request when it has no limit. The author recommends not using CPU limits at all.

    View case
  2. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
    Symptoms

    A tech support engineer called Palark to say that a customer's cluster, production environment included, had practically stopped working. On the control plane nodes the kube-apiserver processes had used up all the memory and were crashing and restarting over and over, so the Kubernetes API was unreachable and completely non-functional.

    Control plane unresponsiveOOMKilled
    Trigger

    Cilium agent pods on more than 200 nodes sent LIST requests to the Kubernetes API en masse at the same time.

    Root cause
    Capacity exhaustion
    Component
    Kubernetes API serverCilium
    Contributing factor
    Unreviewed defaults
    Detection

    A tech support engineer phoned the team to report that the customer's cluster was practically non-operational.

    Remediation
    • added CPU and memory to the control plane nodes in two steps, from 8 CPUs and 16 GB of RAM to 16 CPUs and 64 GB, until the API stabilised
    • restarted the Cilium agents in a test window agreed with the customer to reproduce the memory growth
    • deployed a FlowSchema and a PriorityLevelConfiguration named cilium-pods that limit and queue LIST requests from the Cilium service accounts to cilium.io resources
    • reduced the control plane nodes back to their original resources
    Action items
    • Make an API Priority and Fairness configuration a standard part of Palark's Kubernetes platform
    Lessons

    In a large cluster, many agents sending LIST requests at once can exhaust kube-apiserver memory, and with several API servers the crash moves from one to the next. APF can hold back such a client: a FlowSchema matches its requests and a PriorityLevelConfiguration gives them a small concurrency share and a queue, so the requests run more slowly instead of all at once. With APF enabled, the total concurrency it divides is the sum of --max-requests-inflight (default 400) and --max-mutating-requests-inflight (default 200), shared among priority levels in proportion to their assured concurrency shares, so raising one level's share lowers every other level's limit. Long-running requests such as logs and exec, and WATCH requests, are not subject to APF. The response headers X-Kubernetes-PF-FlowSchema-UID and X-Kubernetes-PF-PriorityLevel-UID show where a request landed, and the metrics Apiserver_flowcontrol_rejected_requests_total, Apiserver_current_inqueue_requests and Apiserver_flowcontrol_request_execution_seconds, with the debug endpoints /debug/api_priority_and_fairness/dump_priority_levels and /debug/api_priority_and_fairness/dump_queues, show how the queues behave.

    View case
  3. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
    Symptoms

    At 14:11 UTC on May 25, 2023, Linode learned that part of its LKE users in the Frankfurt data center could not reach their kube-apiservers through kubectl. The status page opened at 14:27 UTC with "We are investigating reports of connectivity issues with the LKE service within the Frankfurt Datacenter".

    Control plane unresponsive
    Root cause
    Capacity exhaustion
    Component
    Kubernetes API serverContainer memory limit
    Contributing factor
    –
    Detection

    Linode was made aware of the problem at 14:11 UTC; the status page refers to reports of connectivity issues.

    Remediation
    • increased memory limits
    Action items
    • Increase resilience, redundancy and monitoring for the LKE components affected by the incident
    View case
  4. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
    Symptoms

    One evening, right after an ordinary restart of kube-apiserver in a customer's production cluster, etcd memory consumption rose almost threefold. The sudden growth led to a cascading restart of all the master nodes.

    Control plane unresponsiveDatabase overload
    Trigger

    An ordinary restart of kube-apiserver, after which every pod of the log collector DaemonSet sent its list requests at the same time.

    Root cause
    Capacity exhaustion
    Component
    etcdKubernetes API serverTelemetry agent
    Contributing factor
    Unknown limitUnreviewed defaults
    Detection

    Memory graphs showed etcd memory on the master nodes almost tripling after a kube-apiserver restart; the audit logs then pointed to one DaemonSet's list requests.

    Remediation
    • created a PriorityLevelConfiguration named limit-list-custom that queues requests (assuredConcurrencyShares 5, 16 queues, handSize 4, queueLengthLimit 50)
    • created a FlowSchema that routes list and get requests on pods from the log collector's service account to that priority level
    Action items
    • Submit a pull request adding resource version support to the Kubernetes Rust library
    • Open a pull request to the Vector log collector to address the startup list requests
    • Apply the same flow-control approach to a similar problem found with the Cilium CNI
    Lessons

    A field selector reduces what kube-apiserver returns, not what it reads from etcd: listing the pods of one node still reads every pod in the cluster. A DaemonSet whose pods all list at startup multiplies that by the node count each time kube-apiserver restarts. Where exact freshness is not needed, resourceVersion=0 lets kube-apiserver answer from its cache; where the client cannot set it, API Priority and Fairness can queue the requests so they reach etcd gradually.

    View case
  5. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
    Symptoms

    The hub pod of the prod JupyterHub on the leap cluster (leap.2i2c.org) restarted unexpectedly during a large node scale-up, and a user opened a support ticket. The ingress-nginx and proxy pods kept running.

    Pod evictionPods Pending
    Trigger

    A user's dask-gateway workload added about 100 dask-worker nodes within a minute, and Prometheus started scraping the metrics exporters on about 200 nodes.

    Root cause
    Capacity exhaustion
    Component
    PrometheusNode pool
    Contributing factor
    No headroomExcessive blast radiusMonitoring blind spot
    Detection

    A user support ticket (2i2c Freshdesk ticket 414) about the leap hub; the engineer then looked at Grafana and the cluster's nodes and events.

    Action items
    • Configure CPU and memory requests for the hub and proxy pods (#2127; at least 100m CPU was proposed)
    • Make the Grafana OOMKiller dashboard work (#2213)
    • Move LEAP cloud infrastructure to shared nodes (#2209)
    • Update resource requests and limits, mainly for ingress-nginx and Prometheus (#2324)
    • Decide whether Prometheus should scrape dask worker pods (#2366)
    Lessons

    Prometheus load grows with the number of nodes, so a burst of autoscaled nodes can push a nearly full core node over the edge and evict the pods that matter most. Core pods that request 10m of CPU get a tenth of the CPU share of busy neighbours that request 100m. Kubernetes events expire after an hour, so evidence of OOM kills needs a dashboard that keeps it.

    View case
Check your cluster for this

These 51 cases involve Container memory limit and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Capacity exhaustion51 cases

Component
Container memory limit12Node pool11Kubernetes API server9CoreDNS8Application database6kubelet6Prometheus6Telemetry agent6Client-side DNS resolver and cache4etcd4Pod scheduling and placement4In-memory cache3Horizontal pod autoscaler3Ingress controller3VPC subnet or pod address range3Cluster autoscaler2conntrack table2Container runtime2Container CPU limit2Karpenter2kube-dns2kube-proxy2Linux kernel2Node root disk2Persistent volumes and stateful sets2Shared network filesystem2AWS VPC CNI2Admission webhook1CI system1Cilium1Cloud controller manager1Credential broker for pods1Elasticsearch or OpenSearch1GKE control plane1HAProxy1Helm1Istio1Jobs and CronJobs1Language runtime settings1NAT gateway1Cluster authentication provider1Container registry and its credentials1Service discovery1Training data loader1VPC DNS resolver1
Contributing factor
Unreviewed defaults17Monitoring blind spot13Unknown limit12Single point of failure12Retry storm10Automation amplification7Excessive blast radius7Harmful manual action7Unverified assumption6No capacity planning6No alert on limit5No resource requests or limits5Ignored early signal3No caching3No headroom3Unrepresentative staging3Limited debug access2No staged rollout2Ineffective rollback2Change during peak hours1No runbook1Overlapping failures1
Organisation
UK Ministry of Justice52i2c3Zalando3Backpackr2loveholidays2Palark2Scaleway2Adevinta1Airbnb1BC Government1Blue Matador1Buildkite1Civis Analytics1Datadog1DFDS1DigitalOcean1Fullstaq1GitLab1Google Cloud1Honeycomb1Hugging Face1Kevin1Lablup Inc1LINE1Linode1Magic Pages1Mercari1Moonlight1Neon1Onkar Raghunath Shelke1PostHog1RebelMouse1Render1Saifeddine Rajhi1Saltside1Temporal1Universe1unknown1이언철1
  1. IL-0071 Worker node running out of inodes while disk space looked fine, because containerd kept a full node_modules tree in each of several image layer snapshots
    Onkar Raghunath Shelke2026
  2. IL-0102 CoreDNS pods on a production EKS cluster ran out of memory under a burst of new pods, endpoints and nodes, taking down the Buildkite web UI, APIs and job dispatch
    Buildkite2026
  3. IL-0155 Some DOKS clusters in NYC1 saw intermittent DNS timeouts and NodeNotReady events because CoreDNS ran on shared-CPU Droplet worker nodes
    DigitalOcean2026
  4. IL-0137 Training initialisation on a 60-node NVIDIA B200 cluster took more than 8 hours instead of minutes because fragmented small random I/O from all nodes saturated the metadata service of the VAST Data NFS storage
    Lablup Inc2026
  5. IL-0181 A traffic spike from an external community post pushed product API pods into CPU throttling under a 1-core limit, and an e-commerce product page slowed down while every pod stayed Running
    Kevin2026
  6. IL-0129 Throughput of the 384-GPU SmolLM3 pretraining run collapsed because shared WekaFS storage evicted dataset shards to S3, then kept dropping on local NVMe because the nanotron dataloader built an index that grew with the step count
    Hugging Face2025
  7. IL-0100 PostHog's feature flags pods were packed onto Kubernetes nodes that were too small for their CPU requests, and CPU saturation above 90% caused three outages in ten days through Redis overload, connection pool exhaustion and slow queries
    PostHog2025
  8. IL-0048 DNS lookups timed out in a large EKS cluster when CoreDNS hit the 1024 packets-per-second link-local limit, driven by sudo hostname lookups
    unknown2025
  9. IL-0067 Redis filled its persistent volume, corrupted its append-only file and crash-looped on every restart
    BC Government2025
  10. IL-0080 kube-dns on Mercari's GKE clusters could not keep up with peak-hour DNS query rates, and a Sev1 production outage followed when several services failed DNS resolution
    Mercari2025
  11. IL-0082 Losing one CoreDNS pod during a planned EKS node reroll overloaded the remaining CoreDNS pods, causing DNS timeouts and errors across multiple services at RebelMouse
    RebelMouse2025
  12. IL-0006 VPC subnets ran out of IP addresses because the AWS VPC CNI kept IPs on full nodes, and WARM_IP_TARGET changes then blocked pod starts after every pod deletion
    Neon2025
  13. IL-0099 User servers on 2i2c's Earthscope hub took over 15 minutes to start and kernels were killed during a workshop, because a profile option guaranteed 8 GB of memory with no limit on 32 GB nodes
    2i2c2025
  14. IL-0170 During a Lunar New Year event, traffic more than three times normal filled the nf_conntrack table on the EKS node running the Istio ingress gateway, dropping packets and taking down Backpackr's web and app services
    Backpackr2025
  15. IL-0027 All CoreDNS pods landed on one node, the node ran out of memory, and DNS for the whole service went down
    Backpackr2025
  16. IL-0085 A control plane node of Magic Pages' Kubernetes cluster on Hetzner Cloud ran out of memory, the cluster's own calls then used up the Hetzner API rate limit, and all customer websites were down for about 7.5 hours
    Magic Pages2025
  17. IL-0204 An etcd memory spike overloaded the control plane of a Render Frankfurt cluster, and DNS then failed because CoreDNS ran on the rebooted control plane nodes and could not start without the API server
    Render2024
  18. IL-0007 The AWS VPC CNI could not assign IP addresses to pods as an EKS cluster ran out of IPv4 space
    Adevinta2024
  19. IL-0042 Application logs stopped reaching Elasticsearch and OpenSearch for hours and were lost when OpenSearch ran out of disk space
    UK Ministry of Justice2024
  20. IL-0044 Prometheus OOMKilled in a loop while replaying its write-ahead log after a kube-prometheus-stack upgrade
    Saifeddine Rajhi2024
  21. IL-0182 Too few CoreDNS replicas on EKS were overloaded during traffic spikes, raising latency and cutting the network for task pods, and the over-provisioned, tightly spread CoreDNS pods added later blocked Karpenter consolidation
    이언철2024
  22. IL-0046 Prometheus restarted again and again because its WAL replay took longer than the startup probe allowed
    UK Ministry of Justice2024
  23. IL-0110 A Grafana Agent pod's high memory use and crash-looping Hangfire pods drove a Kubernetes node into system OOM and NotReady, twice in one day
    DFDS2024
  24. IL-0168 Simultaneous cluster auto-upgrades overloaded Scaleway's Kubernetes API, and a node authentication retry loop kept Kapsule and Kosmos nodes NotReady in FR-PAR
    Scaleway2024
  25. IL-0169 The Scaleway Kubernetes API in FR-PAR ran out of memory under simultaneous deployments, and because it also handled node authentication, Kapsule and Kosmos nodes went NotReady
    Scaleway2024
  26. IL-0202 A burst of queries and database writes exhausted connection pools and memory on Honeycomb's ingest hosts, and Kubernetes crash-loop back-off kept failed hosts offline so ingest capacity stayed too low to recover
    Honeycomb2023
  27. IL-0043 Application logs missing in Kibana after a newer Fluent Bit version changed its chunk drop strategy
    UK Ministry of Justice2023
  28. IL-0016 Prometheus on the live cluster ran out of memory and was OOMKilled, and the monitoring node it ran on went NotReady
    UK Ministry of Justice2023
  29. IL-0028 Temporal History service latency rose because its pods were CPU-throttled while CPU usage read below 60 percent
    Temporal2023
  30. IL-0195 The kube-apiservers of a cluster with more than 200 nodes ran out of memory and crashed one after another under mass LIST requests from Cilium agents
    Palark2023
  31. IL-0165 Some LKE users in Frankfurt could not reach their kube-apiservers with kubectl after backend services failed under memory constraints
    Linode2023
  32. IL-0194 After a kube-apiserver restart, pod LIST requests from every pod of a Vector DaemonSet nearly tripled etcd memory and restarted all master nodes
    Palark2023
  33. IL-0070 Critical pods on core nodes evicted when Prometheus began scraping about 200 nodes after a dask-gateway scale-up
    2i2c2023
  34. IL-0140 The GKE API in us-central1 returned errors and high latency for 3 hours 19 minutes after an unexpected rise in backend traffic that client retries made worse
    Google Cloud2022
  35. IL-0206 Consul agents in GitLab.com's GKE pods were repeatedly OOM-killed, so the Rails pods lost database replica discovery and sent every query to the primary
    GitLab2022
  36. IL-0094 Student pods filled the shared nodes of a JupyterHub cluster and starved the hub's core pods, freezing the hub
    2i2c2020
  37. IL-0093 Ingress controller pods crash-looped one after another because the leader pod had exhausted its shared memory
    UK Ministry of Justice2020
  38. IL-0176 Prometheus in a 235-node Kubernetes cluster crash-looped because over two million series exhausted host memory and the kernel OOM killer chose it first, as Burstable QoS gave it oom_score_adj 999
    LINE2020
  39. IL-0013 The conntrack table filled up on downsized GKE nodes and a high-traffic aggregation proxy saw climbing connection resets
    loveholidays2020
  40. IL-0015 A GKE cluster's /16 pod address range ran out at 256 nodes, leaving new pods Pending and node scale-ups failing
    loveholidays2020
  41. IL-0033 Uneven load across kube-dns pods and a DNS packets-per-second limit pushed service latency to 20 seconds
    Airbnb2019
  42. IL-0026 Kubernetes nodes hit SystemOOM because fluentd-sumologic log shipper pods had no memory limits
    Blue Matador2019
  43. IL-0002 CI builder pods could not get AWS IAM credentials because kubelet fell minutes behind its API rate limit
    Zalando2019
  44. IL-0076 Kubernetes masters went down after Dex filled etcd with AuthRequest objects and deleting its namespace started an endless list-and-timeout loop
    Fullstaq2019
  45. IL-0037 Image registry rate-limited a whole region after a crash-looping DaemonSet with imagePullPolicy Always pulled through three NAT instances
    Datadog2019
  46. IL-0064 CPU-heavy pods packed onto one Google Cloud node drove it to kernel panics, and the crash moved from node to node until the site was down
    Moonlight2019
  47. IL-0075 Kubernetes API servers pegged on CPU when the cluster grew to about 290 nodes, traced to Datadog agents polling the API
    Civis Analytics2019
  48. IL-0001 Cluster-wide DNS outage after CoreDNS pods were OOM-killed again and again under a burst of DNS queries
    Zalando2019
  49. IL-0031 Kubernetes API server OOM-killed, leaving the Skipper ingress without routes; the talk points to a misconfigured CronJob manifest
    Zalando2018
  50. IL-0036 A reindex Job that kept failing exhausted one node after another until the whole cluster was down
    Universe2017
  51. IL-0040 Parallel Helm installs saturated small gp2 root volumes on a kops cluster and left all three workers NotReady with the kubelet or Docker down
    Saltside2017