Pods stuck Pending after node failure: why didn't Kubernetes reschedule them to the other node?

What we saw: `0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector`

If your pods are not rescheduled when a node goes down and stay stuck Pending after node failure, run kubectl describe pod and read the FailedScheduling event. After a drain it reads 0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable.; after a node dies, the second item is 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: } instead (1 node(s) had untolerated taint(s) on Kubernetes 1.35 and later). Either way, the pod has been recreated, and no node satisfies its own placement rules. On our cluster 15 stateless app Deployments carried a required node affinity, role NotIn [observability], and the role=observability label is on the only other node, so each of them could run on exactly one node. Nine of them also had topology spread rules that kept two defaults: nodeTaintsPolicy (Ignore), which can leave the second replica Pending with didn't match pod topology spread constraints once a node is gone, and matchLabelKeys (empty), which during an ordinary rolling update moved seven single-replica apps to the other node. We changed the rule from requiredDuringSchedulingIgnoredDuringExecution to preferredDuringSchedulingIgnoredDuringExecution with weight 100 and added nodeTaintsPolicy: Honor and matchLabelKeys: [pod-template-hash] to the spread rules; below are the scheduler output, the Kubernetes documents that explain it, read-only checks with the value that means you have the same problem, and the drain test that confirmed the fix. The cluster is a 2-node k3s cluster (k3s v1.33.5+k3s1, one server and one agent); all times are UTC, 2026-11-13 unless a date is given.

The two nodes, both Ready, no taints:

The finding came from a full run of kubegap, an open-source audit of Kubernetes cluster operations, on 2026-11-11 at 05:08:33. Its problem 5, translated from Korean: "The two nodes cannot stand in for each other. The apps are tied to agent-1, and the databases, the registry and the observability stack to server-1, by labels and local-path volumes. server-1 is the only k3s server and etcd, so losing server-1 also stops the API server." It counted 18 workloads that could run only on agent-1 and 24 only on server-1, CoreDNS with 1 replica on server-1 and no PodDisruptionBudget, and both ingress-nginx replicas on server-1.

On 2026-11-13 at 02:25 we listed the node rules of every Deployment and StatefulSet. Fifteen running app Deployments, in four production namespaces, had the same required rule (excerpt, names masked):

$ kubectl get deploy,sts -A -o json | jq …   # a variant of check 1's jq program that also prints replicas and PVC claims; check 1 prints the rule as nodeSelector=… required=…
Deployment   prod-f/signal-f     1  role NotIn observability
Deployment   prod-f/web-f        1  role NotIn observability
Deployment   prod-b/api-b        2  role NotIn observability
Deployment   prod-b/admin-b      1  role NotIn observability
...
Deployment   prod-d/api-d        2  role NotIn observability
Deployment   prod-e/web-e        1  role NotIn observability

The same rule written as a label selector:

$ kubectl get nodes -l 'role notin (observability)'
NAME      STATUS   ROLES    AGE    VERSION
agent-1   Ready    <none>   347d   v1.33.5+k3s1

One node. Nine of the 15 Deployments also had this spread rule, with no nodeTaintsPolicy and no matchLabelKeys:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: api-d

Three of them ran 2 replicas at 02:27, and all six of those pods were on agent-1: api-b, api-d and an admin app that its HorizontalPodAutoscaler later scaled to 1. At 02:35 there were 38 running pods (DaemonSets excluded) on agent-1 and 59 on server-1.

We fixed the rules first (next section) and then drained agent-1 at 02:51:48 to test the fix. So the scheduler never wrote the message at the top of this page for the 15 app Deployments. The scheduler wrote it during the drain for pods that are tied to agent-1 on purpose: a TURN server on hostNetwork, a GPU embedding service and a vector database, each with a kubernetes.io/hostname selector. With the old rule the scheduler would have given the 15 apps' pods the same didn't match Pod's node affinity/selector reason for server-1, because the rule matches only agent-1 (inference from the rule and the selector output above; we did not run the drain before the fix).

At 02:54:12 these eight were the only Pending pods: four pods with local-path volumes on agent-1 (didn't match PersistentVolume's node affinity), the three pods above with hostname selectors, and the second GPU workload (Insufficient nvidia.com/gpu; server-1 has no GPU). server-1 was running 89 non-DaemonSet pods and agent-1 none. One of the eight lines, unmasked except for the pod name:

0/2 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 1 node(s) were unschedulable. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling.

Why

A required `NotIn` rule is satisfied every day and fails on the day the node is lost

The Kubernetes documentation for node affinity:

- requiredDuringSchedulingIgnoredDuringExecution: "The scheduler can't schedule the Pod unless the rule is met."

- preferredDuringSchedulingIgnoredDuringExecution: "If a matching node is not available, the scheduler still schedules the Pod." Each preferred term has a weight from 1 to 100.

- "NotIn and DoesNotExist allow you to define node anti-affinity behavior."

role NotIn [observability] reads as "anywhere except the observability node". With two nodes, it means "only agent-1". IgnoredDuringExecution means running pods are never checked again, so the rule does nothing while agent-1 is up. It is evaluated only when a pod is scheduled, so it fails only when a pod has to be scheduled while agent-1 cannot take it: when agent-1 has failed, is drained or cordoned, or has no room left.

The scheduler runs its filters in a fixed order (NodeUnschedulable, NodeName, TaintToleration, NodeAffinity, …, PodTopologySpread, from default_plugins.go in the Kubernetes source), and a node is rejected by the first filter that fails. A drained node therefore reports were unschedulable. On the surviving node only the node affinity failure is reported; a topology spread failure on the same node is never evaluated, so it does not appear in the message.

For a node that dies instead of being drained, the second item is a taint, not were unschedulable. Assembled from the Kubernetes source, not observed by us: up to Kubernetes 1.34 the line reads 1 node(s) had untolerated taint {node.kubernetes.io/unreachable: }, and from 1.35 it reads 1 node(s) had untolerated taint(s) without the key.

`nodeTaintsPolicy` defaults to Ignore

The Kubernetes documentation for topology spread: "If this value is null, the behavior is equivalent to the Ignore policy." With Honor, "nodes without taints, along with tainted nodes for which the incoming pod has a toleration, are included." The field is GA in 1.33. The Kubernetes blog describes the failure: "This may lead to a node with untolerated taint as the only candidate for spreading, and as a result, the pod will stuck in Pending if it doesn't tolerate the taint." and "For backwards compatibility, nodeTaintsPolicy defaults to Ignore."

What this means on two nodes once the node rule is preferred, as we reasoned it before the test (not reproduced; we added Honor before draining): with Ignore, an unreachable or cordoned agent-1 still counts as a spreading domain, and it holds 0 matching pods, because the spread count skips pods that are terminating (countPodsMatchSelector in the Kubernetes source). The first replacement replica goes to server-1. The second would make the skew 2 − 0 = 2, more than maxSkew: 1, so with DoNotSchedule it stays Pending with didn't match pod topology spread constraints. A 2-replica Deployment would run at half its replicas for as long as agent-1 is gone.

Without `matchLabelKeys`, every rolling update counts the old pod

This part we observed. Our first fix changed the node rule to preferred and added Honor. Argo CD had applied it by 02:39:23, which started a rolling update of all 15 Deployments. The new pod of seven single-replica apps, and one new replica each of api-b and api-d, went to server-1, although the new rule prefers agent-1 with weight 100.

The reason (inference from the spread rule and the documentation below): the labelSelector matches pods of every revision, so during the update the old pod on agent-1 is still counted. A new pod on agent-1 would make the skew 2 − 0 = 2; on server-1 it makes 1 − 1 = 0. DoNotSchedule is a filter and the preferred node affinity is only a score, so agent-1 is removed before scores are compared and the weight of 100 never applies to it.

Those images had never been pulled on server-1. There were nine pulls on server-1 from the in-cluster registry between 02:41:44 and 02:45:35, from 2m25.179s (843,669,709 bytes) to 6m13.859s (835,296,653 bytes); a 94,655,960-byte image took 5m49.087s.

The Kubernetes documentation for matchLabelKeys: with pod-template-hash, "which is added automatically by the Deployment controller", the rule can "distinguish between different revisions in a single Deployment". The field is beta and enabled by default since 1.27. With matchLabelKeys: [pod-template-hash] the old revision's pod is not counted, so a single-replica app's new pod can go back to the preferred node.

What to check in your cluster now

Checks 1 to 4 only read. They need kubectl and jq.

1. Workloads whose required rules leave exactly one node. A required rule (nodeSelector or requiredDuringSchedulingIgnoredDuringExecution) is checked against labels that exist today. List every required node rule with the first command, then rewrite each one as a selector for kubectl get nodes -l (key=value for nodeSelector, key in (a,b) for In, key notin (a) for NotIn, key for Exists, !key for DoesNotExist, a comma for AND, one command per OR term) and count the nodes it prints:

kubectl get deploy,sts -A -o json | jq -r '.items[] | .spec.template.spec as $s
  | select($s.nodeSelector or $s.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution)
  | "\(.kind) \(.metadata.namespace)/\(.metadata.name) nodeSelector=\($s.nodeSelector // {} | tojson) required=\($s.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms // [] | map(.matchExpressions // [] | map("\(.key) \(.operator) \(.values // [] | join(","))") | join(" AND ")) | join(" OR "))"'
kubectl get nodes -l 'role notin (observability)'   # our rule; use yours

For our old rule, kubectl get nodes -l printed one node. Same problem if a stateless Deployment (no local volume, no GPU, no hostNetwork) matches one node: that Deployment cannot start anywhere while that node is down. Workloads pinned on purpose (a database on a local volume, a GPU job) show up here too; for them the answer is a restore, not a reschedule. After our fix, the 27 workloads that still match one node are databases, the registry, observability, a GPU job and a TURN server.

2. Spread rules that block the second replica, or push single replicas off their node.

kubectl get deploy -A -o json | jq -r '
  .items[] | (.metadata.namespace+"/"+.metadata.name) as $n | .spec.replicas as $r |
  .spec.template.spec.topologySpreadConstraints[]? | select(.whenUnsatisfiable=="DoNotSchedule") |
  "\($n) replicas=\($r) key=\(.topologyKey) nodeTaintsPolicy=\(.nodeTaintsPolicy // "Ignore(default)") matchLabelKeys=\((.matchLabelKeys // [])|join(","))"'

Same problem if:

- key=kubernetes.io/hostname, nodeTaintsPolicy=Ignore(default) and replicas 2 or more: when one node is cordoned or unreachable it still counts as a place with 0 replicas, so with maxSkew: 1 only one replica can start on the remaining node.

- the same rule with an empty matchLabelKeys on a Deployment that should stay on one preferred node: during every rolling update the old pod is still counted, and the new pod is forced onto the other node.

After our fix all 9 app rules print nodeTaintsPolicy=Honor matchLabelKeys=pod-template-hash. The k3s packaged CoreDNS Deployment still prints a hostname DoNotSchedule rule with Ignore(default). With 1 replica it does not meet the first condition, and we left it as it is (see "Not changed, and why").

3. Volumes that tie pods to one node.

kubectl get pv -o json | jq -r '.items[] | select(.spec.nodeAffinity.required) |
  "\(.spec.claimRef.namespace)/\(.spec.claimRef.name) \(.spec.storageClassName) \([.spec.nodeAffinity.required.nodeSelectorTerms[].matchExpressions[].values[]]|join(","))"'

Same problem if a production database's PV is listed with one node. Its pod will report didn't match PersistentVolume's node affinity on every other node. Moving it needs a restore from a copy kept elsewhere, or replicated storage. Our local-path PVs: 24 on server-1, 14 on agent-1.

4. Is there room for the pods that would move?

kubectl describe nodes | grep -E '^Name:|^Allocatable:|^Allocated resources:|^  (cpu|memory)[: ]'

Add the Allocated resources requests of the node you might lose to those of the other node, and compare the sum with the other node's Allocatable. This overstates the load that would move, because it includes DaemonSet pods and pods that cannot move; subtract them if the sum comes out close. Ours: the two Allocated resources lines add up to 3,340m + 5,140m = 8,480m CPU, under the 12,000m allocatable, and 15,314 + 17,366 = 32,680 MiB memory, over the 31,452 MiB allocatable. After we subtracted DaemonSet pods and the eight pinned pods, 1,550m CPU and 8,896 MiB would move from agent-1, and server-1 had 6,860m CPU and 14,086 MiB unrequested. Same problem if, after the subtraction, the requests that would move are larger than the other node's Allocatable minus its own requests: some pods will report Insufficient cpu or Insufficient memory.

5. Test it with a drain. This one changes the cluster.

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --timeout=300s
kubectl get pods -A --field-selector status.phase=Pending \
  -o custom-columns='NAMESPACE:.metadata.namespace,POD:.metadata.name,WHY:.status.conditions[0].message'
kubectl uncordon <node>
kubectl rollout restart deployment/<name> -n <namespace>   # pods do not move back on their own

Before draining, let running Jobs finish and look for pods without an owner:

kubectl get pods -A -o json | jq -r '.items[]|select(.metadata.ownerReferences==null)|.metadata.namespace+"/"+.metadata.name'

Drain deletes finished pods, and an ownerless one is gone for good. Exclude it with --pod-selector.

Pass: every Deployment that is not pinned on purpose is Available on the other node, and each Pending pod has a reason you expected (didn't match PersistentVolume's node affinity, Insufficient nvidia.com/gpu, a hostname selector). Fail: a stateless app Pending with didn't match Pod's node affinity/selector or didn't match pod topology spread constraints.

A drain evicts at once. A node that really dies is first marked unreachable, and Kubernetes adds a toleration "with tolerationSeconds=300" to every pod, so pods "remain bound to Nodes for 5 minutes after one of these problems is detected". Add at least five minutes to what the drain shows: the count starts when the node controller marks the node unreachable, not when the node fails.

If you do not want to repeat checks 1 and 3 by hand after every change, kubegap runs them: it reports each workload that can run on only one node, pods tied to one node by a local volume, hostNetwork or a GPU, replicas of one Deployment on the same node, and workloads that depend on a node label. It runs check 4 only for losing the largest node, which for us is server-1, so check 4 for losing agent-1 stays manual. Its report on our cluster had no finding for the spread defaults of check 2, so that one stays manual. Its run on 2026-11-11 is where this problem was found.

What we changed, and what we have not

Changed

Commit 1, pushed 02:38:33. In 16 Deployment manifests (15 deployed, one not deployed), requiredDuringSchedulingIgnoredDuringExecution became preferredDuringSchedulingIgnoredDuringExecution with weight 100 and the same NotIn expression, and nodeTaintsPolicy: Honor was added to the 10 spread rules. A server-side dry-run passed for all 16. Argo CD applied it by 02:39:23.

Commit 2, pushed 02:46:38. matchLabelKeys: [pod-template-hash] on the 10 spread rules. After its rollout settled at 02:50:25, every single-replica app was back on agent-1, and api-b and api-d had one replica on each node.

One Deployment after both commits:

      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          nodeTaintsPolicy: Honor
          matchLabelKeys:
            - pod-template-hash
          labelSelector:
            matchLabels:
              app: api-d
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              preference:
                matchExpressions:
                  - key: role
                    operator: NotIn
                    values:
                      - observability

We chose preferred with weight 100 rather than removing the rule, so that each app keeps its usual node every day and can still start on the other node when its usual node is gone.

The drain test. A backup Job running on agent-1 was allowed to finish first (02:50:47). A probe of 20 public addresses started at 02:50:47, one round about every 5 s. Then:

$ kubectl drain agent-1 --ignore-daemonsets --delete-emptydir-data --pod-selector=app!=<the ownerless pod's app label> --timeout=300s
node/agent-1 cordoned
Warning: ignoring DaemonSet-managed Pods: ...
evicting pod ...
...
node/agent-1 drained
# exit 0 at 2026-11-13T02:52:55Z

The drain started at 02:51:48 and printed 83 eviction lines. The selector excluded an ownerless pod that had been Failed since May, because drain would have deleted it and nothing would have recreated it.

- At 02:52:55 two of the 15 app Deployments were not fully available, admin-b (0 of 1) and api-d (1 of 2); from 02:53:12 to 02:53:51 only api-d was still short, and at 02:54:10 all 15 were fully available. That is 2 min 22 s after the drain started.

- The eight Pending pods were the ones shown in the first section, each with an expected reason.

- 24 containers on server-1 started from images already there and 14 images were pulled, taking from 1.349 s to 2m9.6s. Eight of the nine images pulled during the first rollout were still on server-1; for the ninth, one pod reported it already present at 02:52:00 and another reported a 1.349 s pull at 02:52:02, which we did not look into.

From the probe (an outage window is the first to the last failing round, plus one interval):

- The two 2-replica APIs never left their normal response code during the drain.

- Single-replica apps were off baseline for about 5 to 85 s: 1 round for the search web UI, 4 rounds from 02:51:54 to 02:52:19 for one site, 11 rounds from 02:52:04 to 02:53:02 for one admin app, 15 rounds to 02:53:24 for an internal API that has no node rule.

- A search API returned 503 from 02:52:04 to 02:54:24, because its embedding service needs the GPU on agent-1.

- An API whose Postgres is on a local volume on agent-1 returned 502 and 503 from 02:51:48 to 02:54:51.

Restore. kubectl uncordon agent-1 at 02:54:29; the pinned pods were Ready and nothing was Pending at 02:54:57. Pods do not move back by themselves, so we ran kubectl rollout restart on the 15 app Deployments at 02:54:57 (settled 02:58:50, single-replica apps back on agent-1) and, at 02:59:19, on 11 Deployments without node rules that the drain had moved to server-1. At 03:04:37 there were 37 running pods on agent-1 and 63 on server-1. During the restarts one probe round each returned no answer for two addresses (02:55:18 and 02:56:00), and the search API was off baseline until 03:04:33 while it restarted. All 20 addresses were at their normal code from 03:04:43.

kubegap again. A full rerun at 03:04:58: of the 60 findings the 2026-11-11 report listed under problem 5, 19 are gone (the 15 one-eligible-node findings of the app Deployments and 4 replicas-on-one-node findings) and 41 remain.

Not changed, and why

These are our decisions for this cluster, with the reasons below. They are not advice for your cluster.

- The control plane stays one k3s server on `server-1`. The k3s documentation: "HA embedded etcd cluster must be comprised of an odd number of server nodes for etcd to maintain quorum", and HA needs "Three or more server nodes". Two machines cannot form an etcd quorum that survives one failure, and we will not add a third machine: the cluster runs a few personal projects and small contract work. Losing server-1 is handled as a restore, from the etcd snapshots, server token and database dumps copied to agent-1 twice a day or daily; a restore from those copies has been tested. kubegap's rerun also shows that without server-1 the CPU requests would not fit: 6.0 cores allocatable against 7.61 requested.

- Ingress controller replicas and CoreDNS were not spread. Both run only on server-1 (ingress-nginx: 2 replicas, MetalLB L2 address, externalTrafficPolicy: Local, PodDisruptionBudget minAvailable 1; CoreDNS: 1 replica, no PodDisruptionBudget). Both matter only when server-1 is lost, which is the restore case above. CoreDNS is a k3s packaged component; the k3s documentation says those manifests "should not be altered. The files are re-written to disk whenever K3s is started", so changing it would also need a .skip file on the server so that k3s stops managing it.

- Pods tied to `agent-1` by local-path volumes, hostNetwork or GPUs stay tied. Moving them needs replicated storage, a network change or another GPU.

Where else this happened

The Incident Library, a searchable collection of public post-incident reports, has no case with this exact cause (a required node affinity or a topology spread default leaving replacement pods Pending after a node loss). These three show a cluster that could not survive losing nodes: replicas packed onto one node, a control plane node that could not be replaced, and all workers NotReady at once.

- Moonlight, 2019-01-18: on a three-node GKE cluster the scheduler sometimes put all three replicas of a web app on one node; CPU-heavy pods drove that node into kernel panics, and the crash moved from node to node until the site was down. Anti-affinity rules on all major Deployments restored it.

- Magic Pages, 2025-04-14: one of three control plane nodes of a hetzner-k3s cluster ran out of memory, and the cluster's own Hetzner API calls kept the rate limit at zero, so the node could not be replaced and every customer site went down.

- Saltside, 2017-04-26: parallel Helm installs saturated small gp2 root volumes and left all three workers of a kops cluster NotReady.

The three cases above are part of a set of six in which a NotReady node was a single point of failure: Node NotReady + single point of failure, 6 cases. Two of the cases under Kubernetes API server + single point of failure, 7 cases are clusters that went down with their only control plane node, as ours would if server-1 were lost: a homelab whose only master node kernel-panicked after a power outage (2025-03-03), and NRE Labs, whose single-master kubeadm cluster kept losing its API server (2018-12-04).

Timeline

Frequently asked questions