Incident Library

IL-0442 LIFULL · 2021-09-22 (published)kubernetesaws

Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked

KEEL runs most of its pods on AWS spot instances, and the team had added a spot fleet, a fallback for when spot instances are sold out and safe termination for pods on spot instances. Scheduling waits were still expected, so the team introduced PriorityClasses to order pods: low (value: 100) for pods that may be preempted, medium (globalDefault: true, value: 1000) for standard pods and high (value: 10000) for important pods. Preemption is an ordinary pod termination run by kube-scheduler: preStop, then SIGTERM, then SIGKILL if the pod has not stopped after terminationGracePeriodSeconds.

A pod's priority defaults to 0 and is set by the admission controller when the pod is created, so every pod deployed before the PriorityClasses kept priority: 0. With medium as the global default, newly deployed pods got priority 1000, and when they landed on nodes without enough free resources the scheduler preempted the older priority 0 pods all at once, despite their PodDisruptionBudgets.

KEEL then settled on PriorityClasses low (value: -10), medium (value: 0, globalDefault: true) and high (value: 10), each with a -nonpreempting variant that sets preemptionPolicy: Never. Because the global default has value 0, adding PriorityClasses later cannot preempt existing pods. The lower bound is -10 because cluster-autoscaler ignores pods whose priority is below --expendable-pods-priority-cutoff, whose default is -10. Two weeks before this failure the team had shared a Grafana Labs postmortem of a similar failure, and the engineer who shared it was the one who introduced the PriorityClasses.

Severity
–
Scope
Requests from outside the KEEL cluster failed at a sharply higher rate while istio-ingressgateway and many other pods restarted; services were briefly down
Users affected
–
Region
–
Time to detect
–
Time to resolve
–

Symptoms

Shortly after the KEEL team applied new PriorityClasses to KEEL, LIFULL's Kubernetes cluster, and saw pods with the low class being preempted as intended, the team received a flood of alerts from external monitoring: the error rate of requests coming from outside the cluster had risen sharply.

The investigation found that many pods, starting with istio-ingressgateway, the reverse proxy that accepts requests from outside the cluster, showed signs of having restarted. Requests inside the cluster were retried by Istio, the service mesh KEEL uses, so they were barely affected, but with fewer istio-ingressgateway pods the error rate of external requests jumped.

Almost every pod on KEEL had a PodDisruptionBudget, which the team expected to prevent this kind of mass disruption. It did not, because the scheduler respects PodDisruptionBudgets during preemption only on a best-effort basis and still preempts lower-priority pods when it finds no victims that keep their budgets.

Symptom Pod evictionHigh error rate

Root cause

Root cause Configuration change

Component Pod priority and preemptionIngress controller
Contributing factor Unverified assumption
Trigger
The KEEL team deployed three PriorityClasses to the running cluster: low with value: 100, medium with globalDefault: true and value: 1000, and high with value: 10000.

Detection

External monitoring sent the team a large number of alerts about a sharp rise in the error rate of requests from outside the cluster.

Remediation

–

Action items

Lessons

Adding PriorityClasses to a running cluster needs care because every existing pod has priority 0, and a globalDefault class above 0 makes all pods created before it candidates for preemption. PodDisruptionBudgets do not reliably protect pods from preemption. In a multi-tenant cluster, the permission to deploy PriorityClasses should be restricted, and the PriorityClasses tenants may use should be controlled with ResourceQuota or OPA Gatekeeper.

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Pod priority and preemption, Ingress controller among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.