Incident Library
Applied filters Root causeDependency failureComponentNode operating system imageFactorMonitoring blind spotClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke
    FREE NOW2019
    Symptoms

    At 16:00 CEST a developer reported in the #ask-platform Slack channel that a deployment to the Prelive cluster was failing, and a second report followed at 16:06.

    Pods PendingScale-up failure
    Trigger

    Roughly two to three days earlier, the CentOS repository mirror was restructured and container-selinux-2.68-1.el7.noarch.rpm moved, which made the static download URL hardcoded in kops nodeup invalid.

    Root cause
    Dependency failure
    Component
    Node operating system imageNode pool
    Contributing factor
    Monitoring blind spot
    Detection

    Developers asked for help in the #ask-platform Slack channel because their deployments were failing.

    Remediation
    • downloaded the package by hand on the Prelive nodes so their bootstrap could finish
    • changed the cluster AMI from Amazon Linux 2 to Debian and applied it to all clusters
    Action items
    • Create an alert that shows when nodes cannot join the cluster, possibly from information the cluster autoscaler already has
    • Document the external dependencies of node startup and look for ways to reduce them
    • Change the AMI back to Amazon Linux 2 Minimal once kops fixes the issue
    • Document node troubleshooting better: what happens during bootstrap and where the logs are
    Lessons

    Applying cluster changes through kops and Jenkins worked well, and troubleshooting quickly showed that the failing deployments came from missing cluster capacity. What went wrong was that nobody noticed, over a long period, that some autoscaling groups held more worker instances than Kubernetes had nodes. The team was lucky that no large number of production spot instances was terminated during the problem and that it was noticed during office hours.

    View case
Check your cluster for this

This case involves Node operating system image and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
Node operating system imageNode pool
Contributing factor
Monitoring blind spot
Organisation
FREE NOW
  1. IL-0039 New Kubernetes worker nodes could not join any cluster because a package URL hardcoded in kops nodeup broke
    FREE NOW2019