Incident Library

IL-0467 Kubernetes project · 2019-03-26kubernetesgcp

Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours

Boskos is the service in the Kubernetes project's Prow test infrastructure that hands out GCP projects to test jobs. A commit that landed in December 2018 (the timeline gives 12-05 16:21) contained a bug that kept Boskos from starting: it tried to create its resources.boskos.k8s.io CRD with a definition that could never be valid. On 2019-03-20 the Boskos StatefulSet spec was updated to an image with this bug, but the running pod was not replaced, because the StatefulSet had no configuration that rolled out updates, and nobody noticed that the deploy had changed nothing. Unlike most of Prow, Boskos was deployed by hand.

At around 17:20, several hours before the outage, the team began upgrading the Prow clusters from Kubernetes 1.11 to 1.13 from the GKE console. The build cluster has 150 nodes and each node update takes five minutes, so the whole upgrade would take about ten hours. Nothing looked wrong several hours in, and the team went to bed. At 23:04 the node holding the only Boskos pod was drained to be replaced with a 1.13 node. The new pod was created from the StatefulSet's spec with the buggy image and crash-looped. Every job that needed a Boskos project failed for just over nine hours while on-call was asleep. Users reported the failure from 02:50, and the monitoring dashboard showed it, but no alert caught it.

An engineer woke at 07:16 and saw the Slack reports, and at 07:37 another engineer traced the failure to the December commit. A new image was built and deployed, and the timeline records a fixed Boskos running at 08:10 as the end of the outage. Boskos recovered at once, the projects left open were cleaned up over the next few minutes, and tests passed again shortly after. The audit logs then confirmed that the pod had run gcr.io/k8s-testimages/boskos:v20180405-12e892d69 until the drain and that the update of the week before had had no effect.

Severity
–
Scope
All Prow test jobs that needed a GCP project from Boskos failed for just over nine hours, including the required pull-kubernetes-e2e-gce job, so no kubernetes/kubernetes pull request could be merged; 70 failures on pull-kubernetes-e2e-gce alone, with similar effects on other projects
Users affected
Contributors to kubernetes/kubernetes and other projects whose test jobs used Boskos
Region
–
Time to detect
3h 46m
Time to resolve
9h 6m

Symptoms

Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

The jobs failed with "Something went wrong: failed to prepare test environment: --provider=gce boskos failed to acquire project: Post http://boskos.test-pods.svc.cluster.local./acquire?type=gce-project&state=free&dest=busy&owner=pull-kubernetes-e2e-gce", and the request itself ended in "dial tcp 10.63.250.132:80: i/o timeout". The only Boskos pod was in a crash loop, logging the fatal error "unable to create a CRD client" with "CustomResourceDefinition.apiextensions.k8s.io \"resources.boskos.k8s.io\" is invalid: [spec.versions: Invalid value: []apiextensions.CustomResourceDefinitionVersion(nil): must have exactly one version marked as storage version, status.storedVersions: Invalid value: []string(nil): must have at least one stored version]".

The failures began at 23:04, when the node running Boskos was drained during an upgrade of the Prow clusters from Kubernetes 1.11 to 1.13 that had been started that evening, and on-call was asleep. Users reported it on GitHub at 02:50 and on Slack at 05:16 and 06:27. The Boskos monitoring dashboard showed the failure, but no alert fired. The pull-kubernetes-e2e-gce job alone recorded 70 failures from the outage.

The engineers first suspected the Kubernetes upgrade, perhaps a change between 1.11 and 1.13 in how an invalid CRD is handled, although the timing did not quite match. The CRD that Boskos tried to create would never have been valid in any version. The cause was a Boskos code change, and the audit logs showed that the pod had kept running the old image gcr.io/k8s-testimages/boskos:v20180405-12e892d69 until the drain, even though the StatefulSet had been changed the week before.

Symptom Container crash loopService outage

Root cause

Root cause Faulty deployment

Component CI systemNode pool
Contributing factor Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours
Trigger
At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

Detection

Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

Remediation

Action items

Lessons

The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

Sources

References

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CI system, Node pool among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.