Incident Library
Applied filters ComponentNode poolSymptomContainer crash loopFactorChange started before off-hoursClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019
    Symptoms

    Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

    Container crash loopService outage
    Trigger

    At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

    Root cause
    Faulty deployment
    Component
    CI systemNode pool
    Contributing factor
    Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours
    Detection

    Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

    Remediation
    • Built and deployed a new Boskos image with the bug fixed
    • Cleaned up the GCP projects that had been left open
    Action items
    • Change the Boskos configuration so that deployments actually take effect (#11956)
    • Automate Boskos deploys (#11957)
    • Improve alerting on Boskos status (#11953)
    Lessons

    The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

    View case
Check your cluster for this

This case involves CI system and Node pool. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Faulty deployment1 case

Component
CI systemNode pool
Contributing factor
Unverified assumptionChange started before off-hoursChange with a delayed effectNo alert on limitSingle point of failure
Organisation
Kubernetes project
  1. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours
    Kubernetes project2019