Cases2 cases

1 to 2 of 2

1. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release

   Gitpod2022

   Symptoms

   From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

   Throughput degradationSustained high node CPUNode NotReadyData loss

   Trigger

   The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

   Root cause

   Faulty deployment

   Component

   Node root diskContainer CPU limit

   Contributing factor

   No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access

   Detection

   Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

   Remediation

   • Shifted workspace starts back to the ws38 clusters on April 11th
   • Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
   • Requested a limit increase from Let's Encrypt
   • Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
   • Cleaned up workspaces stuck in a stopping state so users could start them again
   • Released ws40 with 1TB boot disks to get more IO bandwidth
   • Removed the workspaces that consumed excessive CPU time for crypto-mining
   • Blocked the fraudulent users behind the crypto-mining from using Gitpod
   • Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
   • Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
   • Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step

   Action items

   • Invest in test automation so that test results stop releases like this one
   • Vary the interval between traffic shifts to new clusters, with more time for larger releases
   • Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
   • Reduce the impact teams have on each other
   • Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
   • Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
   • Make the status page show the workspace service as down when it is, and update the playbooks
   • Keep incident communication in a single Slack channel and Google Meet
   • Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability

   Lessons

   Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

   View case
2. IL-0401 Prow jobs on the k8s-prow-build GKE cluster failed and merges stalled as nodes hit IOPS throttling and kernel panics

   Kubernetes project2019

   Symptoms

   On 30 May 2019, in the run-up to the Kubernetes 1.15 code freeze, Prow test jobs in the k8s-prow-build cluster failed in surprising ways or were aborted partway through. Jobs were reported to GitHub as failed while Deck still showed them as running, because pods exited without uploading `finished.json`, the file most Prow tooling uses to tell a job's state.

   Node NotReadyNode kernel panicPods stuck in TerminatingThroughput degradation

   Trigger

   A spike in test load before the Kubernetes 1.15 code freeze pushed disk I/O on the build nodes far beyond the IOPS that GCP allows for 250 GB disks.

   Root cause

   Capacity exhaustion

   Component

   Cloud block storageNode root diskkubeletCI system

   Contributing factor

   Monitoring blind spotIgnored early signal

   Detection

   Several users reported jobs that GitHub showed as failed but Deck showed as still running, and the first report came on Slack at 13:26. Alerting for these failures did not exist or did not work.

   Remediation

   • Requested and received additional disk quota
   • Brought up 55 nodes with 500 GB disks in a new node pool
   • Migrated all nodes from the 250 GB disk pool to the 500 GB disk pool by draining old nodes, finishing at 23:55

   Action items

   • Create or fix monitoring so the oncall is paged when tide stops doing things
   • Set a process and guidelines for adding new members to the oncall rotation, with preparation and extra support

   Lessons

   Prow metrics were not recorded before the incident, which made it hard to understand, and alerting for these failures did not exist or did not work. After the cause was found and a fix devised, nothing happened for another two hours. The team counts it as luck that someone who recognised IOPS as the problem happened to be around to look.

   View case

By root cause2