Incident Library
Applied filters Root causeSoftware defectOrganisationLinodeClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
    Symptoms

    From about 01:00 UTC on 2025-03-05, Linode Kubernetes Engine (LKE) clusters with IP ACL enabled failed to autoscale: new nodes could not join the cluster, and some customers had service disruptions. About 20 to 25 clusters in several regions around the world were affected; the status page lists the US-IAD (Washington), US-ORD (Chicago), US-SEA (Seattle) and IN-MAA (Chennai) LKE components.

    Scale-up failureService outage
    Trigger

    Akamai deployed the latest LKE software release.

    Root cause
    Software defect
    Component
    Cluster autoscaler
    Contributing factor
    –
    Detection

    Customers began having problems at about 01:00 UTC, and the first status post, at 06:49 UTC, said the team was investigating an LKE autoscaling problem when IP ACL is turned on.

    Remediation
    • restarted the gateway pods, which gave only short-term relief
    • rolled back the release on all affected clusters
    Action items
    • Continue investigating the root cause and take preventive actions
    Lessons

    The defect hit only clusters with IP ACL enabled, and it showed when the autoscaler tried to add nodes. Restarting the gateway pods did not hold; rolling back the release ended the incident.

    View case
  2. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes
    Linode2024
    Symptoms

    At about 14:05 UTC on 22 April 2024, Linode's team noticed that new LKE clusters were failing to provision in the Toronto data center.

    Cluster or node provisioning stallIntermittent timeouts
    Trigger

    A previous update to the API did not work as intended and caused timeouts when new LKE clusters were provisioned.

    Root cause
    Software defect
    Component
    Node pool
    Contributing factor
    –
    Detection

    Linode's team noticed new LKE clusters failing to provision in the Toronto data center at about 14:05 UTC on 22 April 2024.

    Remediation
    • disabled the affected section of Toronto infrastructure from provisioning new LKE clusters
    • implemented a fix for the faulty API update
    • put the disabled infrastructure back into use after the fix
    Lessons

    The fault was in an earlier API change, not in the infrastructure that showed it, but most failures concentrated on one backend resource in one data center. Taking that resource out of provisioning limited the impact while the root cause was found, which took three days.

    View case
Check your cluster for this

These 2 cases involve Cluster autoscaler and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Cluster autoscaler1Node pool1
Contributing factor
–
Organisation
Linode2
  1. IL-0157 After a release, new nodes could not join Linode Kubernetes Engine clusters with IP ACL enabled, so autoscaling failed on about 20 to 25 clusters
    Linode2025
  2. IL-0161 A faulty earlier API update made new LKE cluster provisioning in Toronto time out for three days, leaving clusters with no node pools or the wrong number of nodes
    Linode2024