Incident Library
Applied filters Root causeFaulty deploymentComponentCI systemFactorNo resource requests or limitsClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019
    Symptoms

    On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.

    Pods PendingSustained high node CPUHigh latency
    Trigger

    At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.

    Root cause
    Faulty deployment
    Component
    CI system
    Contributing factor
    Unrepresentative stagingNo resource requests or limitsLimited debug access
    Detection

    While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.

    Remediation
    • Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
    • Applied the revert by hand with kubectl apply after the deployment permission was missing
    • Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
    • Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
    • Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy
    Action items
    • Find out why the bug did not show deterministically
    • Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow
    Lessons

    The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.

    View case
Check your cluster for this

This case involves CI system. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
CI system
Contributing factor
Limited debug accessNo resource requests or limitsUnrepresentative staging
Organisation
Kubernetes project
  1. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019