Incident Library
Applied filters FactorUnverified assumptionOrganisationLIFULLClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0442 Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked
    LIFULL2021
    Symptoms

    Shortly after the KEEL team applied new PriorityClasses to KEEL, LIFULL's Kubernetes cluster, and saw pods with the low class being preempted as intended, the team received a flood of alerts from external monitoring: the error rate of requests coming from outside the cluster had risen sharply.

    Pod evictionHigh error rate
    Trigger

    The KEEL team deployed three PriorityClasses to the running cluster: low with value: 100, medium with globalDefault: true and value: 1000, and high with value: 10000.

    Root cause
    Configuration change
    Component
    Pod priority and preemptionIngress controller
    Contributing factor
    Unverified assumption
    Detection

    External monitoring sent the team a large number of alerts about a sharp rise in the error rate of requests from outside the cluster.

    Action items
    • Redefine the PriorityClasses as low (-10), medium (0, globalDefault) and high (10), each with a non-preempting variant
    Lessons

    Adding PriorityClasses to a running cluster needs care because every existing pod has priority 0, and a globalDefault class above 0 makes all pods created before it candidates for preemption. PodDisruptionBudgets do not reliably protect pods from preemption. In a multi-tenant cluster, the permission to deploy PriorityClasses should be restricted, and the PriorityClasses tenants may use should be controlled with ResourceQuota or OPA Gatekeeper.

    View case
Check your cluster for this

This case involves Ingress controller and Pod priority and preemption. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Configuration change1 case

Component
Ingress controllerPod priority and preemption
Contributing factor
Unverified assumption
Organisation
LIFULL
  1. IL-0442 Older pods on LIFULL's KEEL Kubernetes cluster were preempted all at once and the external error rate spiked
    LIFULL2021