Cases3 cases

1 to 3 of 3

1. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

   Kubernetes project2019

   Symptoms

   On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

   Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

   Trigger

   A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageNode root diskkubeletCI system

   Contributing factor

   Monitoring blind spotIgnored early signal

   Detection

   Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

   Remediation

   • Requested and received additional disk quota
   • Brought up 55 nodes with 500 GB disks in a new node pool
   • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55

   Action items

   • Create or fix monitoring so the oncall is paged when tide stops doing things
   • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support

   Lessons

   Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

   View case
2. IL-0467 Boskos crash-looped after a node drain started it from a buggy StatefulSet image, failing Kubernetes PR jobs for 9 hours

   Kubernetes project2019

   Symptoms

   Overnight into 26 March 2019 (PDT), every Prow test job that needed a GCP project from Boskos, the Kubernetes test-infra service that hands out GCP projects to test jobs, failed. The required job pull-kubernetes-e2e-gce failed on every run, so no pull request could be merged in kubernetes/kubernetes for just over nine hours, and other projects saw similar failures.

   Container crash loopService outage

   Trigger

   At 23:04 PDT the node running the only Boskos pod was drained to be replaced with a 1.13 node during the upgrade of the Prow clusters from Kubernetes 1.11 to 1.13. The replacement pod started from the StatefulSet's current spec, which named a newer Boskos image containing the bug.

   Root cause

   Faulty deployment

   Component

   CI systemNode pool

   Contributing factor

   Change with a delayed effectUnverified assumptionSingle point of failureNo alert on limitChange started before off-hours

   Detection

   Users reported the failure on GitHub at 02:50 and on Slack at 05:16 and 06:27, and an engineer saw the Slack reports on waking at 07:16. The Boskos monitoring dashboard showed the failure, but no alert caught it.

   Remediation

   • Built and deployed a new Boskos image with the bug fixed
   • Cleaned up the GCP projects that had been left open

   Action items

   • Change the Boskos configuration so that deployments actually take effect (#11956)
   • Automate Boskos deploys (#11957)
   • Improve alerting on Boskos status (#11953)

   Lessons

   The write-up lists what went poorly: a ten-hour upgrade was started shortly after 17:00 when on-call coverage ended at 18:00; deployments were not checked to confirm that something had actually rolled out; the Boskos StatefulSet was not set up to perform updates, so changing its spec left the old pod running; the December change appears never to have been tested; and Boskos was deployed by hand, unlike most of Prow, which left more room for error. Once people were available, the problem was fixed quickly.

   View case
3. IL-0466 Prow scheduled test job pods onto its service cluster, leaving hundreds Pending and starving Prow's own services

   Kubernetes project2019

   Symptoms

   On 8 February 2019 the Kubernetes project's Prow instance at prow.k8s.io became very slow: its controllers took about five minutes to respond to the hook, bot commands such as the cat command stopped working, and the prow.k8s.io site was hard to reach. GitHub status was normal, and the Prow service nodes were found running above 100% of CPU and memory.

   Pods PendingSustained high node CPUHigh latency

   Trigger

   At 13:20 PST an auto-generated bump PR for Prow was merged and deployed, and Prow started scheduling workloads onto the service cluster.

   Root cause

   Faulty deployment

   Component

   CI system

   Contributing factor

   Unrepresentative stagingNo resource requests or limitsLimited debug access

   Detection

   While working on a PR, a maintainer noticed the bot's cat command failing, found GitHub status normal, learned on Slack that the Prow nodes were above 100% CPU and memory, and then found 600 pending pods in the test-pods namespace.

   Remediation

   • Rolled Prow back to v20190206-b01b288 and cleaned up the pods and ProwJobs in the service cluster
   • Applied the revert by hand with kubectl apply after the deployment permission was missing
   • Sent a fix, PR 11209, so the client no longer overwrites a provided literal default client
   • Rolled Prow back to v20190125-2aca69d, from before the bug was introduced
   • Cleaned up the wrongly scheduled pods and ProwJobs again and confirmed Prow was healthy

   Action items

   • Find out why the bug did not show deterministically
   • Be more careful with sweeping Prow changes, with more end-to-end tests or a real staging Prow

   Lessons

   The behaviour was not deterministic, and the change was deployed on a Friday afternoon. Prow instances are set up differently, for example in how build clusters are defined, so another project's Prow running a version fine does not show it is safe for the Kubernetes Prow, which needs a staging environment that looks like it.

   View case

By root cause2