Incident Library
Applied filters Root causeFaulty deploymentComponentNode poolSymptomUpgrade stallClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases3 cases

1 to 3 of 3

  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
    Symptoms

    At 09:36 on 18 March 2025, during the upgrade of the live cluster to EKS 1.30, users reported that their pods were being terminated and were not restarting automatically, in both production and non-production environments. Automated infrastructure alerts fired as well, and production services had outages.

    Pod evictionPods PendingService outageUpgrade stall
    Trigger

    Upgrading the live EKS cluster from 1.29 to 1.30, during which the descheduler attempted to reschedule all pods on the cordoned nodes simultaneously.

    Root cause
    Faulty deployment
    Component
    DeschedulerNode poolCluster autoscaler
    Contributing factor
    No headroomNo alert on limitUnreviewed defaults
    Detection

    Users reported at 09:36 that pods were being terminated and not restarting, and automated infrastructure alerts fired.

    Remediation
    • Removed the cordon annotation so pods could be rescheduled
    • Deleted stuck 1.30 nodes to force their recreation
    • Added a new node group of 20 nodes running 1.30 by hand
    • Cordoned and drained all old 1.29 nodes onto the new 1.30 nodes
    Action items
    • Raise an AWS Production System Down support case (Case ID: 174229643100496)
    • Get guidance from AWS support on querying control-plane logs to find the root cause of similar issues
    • Document a runbook for emergency scaling of a new node group in case of unexpected capacity shortages
    • Alert on an imbalance between the autoscaling group and the Kubernetes nodes
    • Review the descheduler configuration to understand why it throws pods off cordoned nodes, and fix it where possible
    • Alert on spikes in the pending pod count
    Lessons

    The write-up lists review actions rather than lessons. They show the team had no documented way to add an emergency node group, no alert on a mismatch between the autoscaling group and the nodes Kubernetes sees, no alert on a spike in pending pods, and no clear understanding of how the descheduler treats cordoned nodes.

    View case
  2. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently
    Linode2022
    Symptoms

    From about 18:30 UTC on June 15, 2022, customers of the Linode Kubernetes Engine (LKE) who tried to upgrade a cluster or add nodes saw those operations fail silently, without an error. The status page opened an investigation of an LKE issue at 19:54 UTC, and the affected list covered LKE in eleven regions.

    Upgrade stall
    Trigger

    An incremental update pushed to the LKE production environment at about 18:30 UTC.

    Root cause
    Faulty deployment
    Component
    Node pool
    Contributing factor
    –
    Detection

    The status page began investigating at 19:54 UTC, 1 hour 24 minutes after the update.

    Remediation
    • Released a follow-up LKE update at about 21:00 UTC that contained the missing changes
    Action items
    • Add steps and automation to the update process
    View case
  3. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019
    Symptoms

    From 07:20 UTC on 28 September 2019, customers running Azure Kubernetes Service clusters on Virtual Machine Scale Sets (VMSS) got failures when they tried any cluster upgrade. The error read: "The platform image 'microsoft-aks:aks:aks-ubuntu-1604-201909:2019.08.21' is not available. Verify that all fields in the storage profile are correct." The issue title calls the error PlatformImageNotFound.

    Upgrade stall
    Trigger

    A recent service update affected instances of a backend service, which became unhealthy and stopped requests from completing.

    Root cause
    Faulty deployment
    Component
    Node operating system imageNode pool
    Contributing factor
    –
    Detection

    Customers received failure messages when they tried to upgrade their clusters.

    Remediation
    • published az vmss update commands that set the scale set's image version to 2019.09.25 and SKU to aks-ubuntu-1604-201909, after which the upgrade could be run again
    • mitigated affected clusters by hand (AKS and VMSS teams)
    • rolled out hot fixes to all regions, completed by 15 October 2019
    View case
Check your cluster for this

These 3 cases involve Node pool and Cluster autoscaler. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment3 cases

Component
Node pool3Cluster autoscaler1Descheduler1Node operating system image1
Contributing factor
Unreviewed defaults1No alert on limit1No headroom1
Organisation
Linode1Microsoft Azure1UK Ministry of Justice1
  1. IL-0020 During an EKS 1.30 upgrade the descheduler tried to reschedule every pod on the cordoned nodes at once and production workloads went down
    UK Ministry of Justice2025
  2. IL-0164 An incremental Linode Kubernetes Engine production update that was missing changes from the previous release made cluster upgrades and node additions fail silently
    Linode2022
  3. IL-0149 Upgrades of AKS clusters on Virtual Machine Scale Sets failed with a 'platform image is not available' error after a service update left a backend service unhealthy
    Microsoft Azure2019