Incident Library
Applied filters Root causeFaulty deploymentComponentCI systemClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases6 cases

6 to 6 of 6

  1. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019
    Symptoms

    On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.

    Pods PendingSustained high node CPUHigh latency
    Trigger

    At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.

    Root cause
    Faulty deployment
    Component
    CI system
    Contributing factor
    Unrepresentative stagingNo resource requests or limitsLimited debug access
    Detection

    While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.

    Remediation
    • Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
    • Applied the revert by hand with kubectl apply after the deployment permission was missing
    • Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
    • Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
    • Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy
    Action items
    • Find out why the bug did not show deterministically
    • Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow
    Lessons

    The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.

    View case
Check your cluster for this

These 6 cases involve CI system and Database. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment6 cases

Component
CI system6Database3In-memory cache1ConfigMaps and Secrets1Helm1Network control plane (SDN)1Node pool1Container registry and its credentials1
Contributing factor
Excessive blast radius3Unverified assumption2Change with a delayed effect2Signal buried in noise1Change started before off-hours1Limited debug access1No alert on limit1No configuration tests1No resource requests or limits1No staged rollout1Overlapping failures1Missed in review1Single point of failure1Unrepresentative staging1Unsafe failure behaviour1Dependence on a warm cache1
Organisation
Kubernetes project2CircleCI1IONOS Cloud1Railway1Wikimedia Foundation1
  1. IL-0356 Every domain hosted on Railway returned HTTP 404 to new connections for five minutes because route lookups failed
    Railway2026
  2. IL-0234 An onboarding operator deployed without its scope file installed dev applications on other IONOS Kubernetes clusters
    IONOS Cloud2026
  3. IL-0324 Three MediaWiki deployments lost Memcached after a PHP 8.1 rollout and overloaded the External Storage database cluster
    Wikimedia Foundation2025
  4. IL-0552 CircleCI's archival database ran out of storage, slowing workflow and job starts and the display of workflows in the UI
    CircleCI2025
  5. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019
  6. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services
    Kubernetes project2019