Incident Library

IL-0591 Gitpod · 2022-04-11kubernetesgcp

Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release

Gitpod released ws39, a new generation of its workspace clusters, on April 8th and 9th, 2022, to cut workspace startup times and give users more CPU. The release came 3.5 weeks after the previous one, and ws39 replaced ws38 within about a day. Several of its changes combined. The 1TB disk was split into many smaller RAID0 disks to separate IO for image pulls and workspaces, but the IOPS a disk can reach decrease with disk size, so disk IO was slower than expected. An enhancement to CPU limiting had a bug that let too many noisy neighbours through, and their effect grew because faster machine types and the removal of slirp4netns from workspace networking made more CPU available to workspaces. The workspace networking changes also had defects that slowed network processes, including containerized workloads. Gitpod had no IO limiting at the time.

On April 11th workspaces became IO bound. Gitpod moved workspace starts back to ws38 by 15:37 UTC and then to new ws39a clusters with c2d machines and local SSDs by 21:29 UTC. Building ws39a meant deploying to a new region because c2d was not available in the usual zones, waiting for a slow GCP quota approval, and getting past a Let's Encrypt rate limit that a customer's certificate requests had used up for gitpod.io. On April 12th workspaces were IO bound again, and the new IO limiter rolled out with ws-daemon broke the CPU limiter on ws-eu39xl, made its nodes unresponsive and lost users' data. Because a breaking control plane API change had made ws38 incompatible, EU XL traffic went to the regional EU cluster while the replacement cluster ws-eu39bxl was built. ws40, with 1TB boot disks and a rewritten IO limiter, took all traffic on April 14th, but nodes still saturated their IO.

On April 17th and 18th the broken CPU limiting let crypto-currency miners, most of them on auto-generated GitHub accounts, take over many workspace nodes, and the team removed their workspaces and blocked them by hand. At 23:35 UTC on April 18th the team found the main issue in the dynamic CPU limiting and fixed it by rolling back a ws-daemon change that ws38 did not have; the fix shipped in clusters named ws34, an old name used because of the Let's Encrypt limit, and was fully rolled out by 16:34 UTC on April 19th. IO limiting built on a library chosen after many experiments went out with ws41xl from 23:48 UTC on April 20th, and traffic moved gradually until the XL clusters were at 100% at 18:46 UTC on April 21st and the standard regional clusters at 02:00 UTC on April 22nd. Gitpod's review says the April 8th changeset lacked risk assessment and was released too quickly, test results should have stopped it, and committing to a rollback on April 11th would have been best.

Severity
–
Scope
Gitpod workspaces: sustained performance degradation, failed workspace starts at times, and data loss for users on the ws-eu39xl cluster
Users affected
Gitpod users in all regions, including unleashed customers and Gitpod staff on the EU XL cluster
Region
all regions
Time to detect
–
Time to resolve
–

Symptoms

From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

The degradation, which affected workspaces in all regions, came back again and again between April 11th and 22nd. After traffic moved to new ws39a clusters with faster machines and local SSDs, workspaces were IO bound again on April 12th. On April 14th, after traffic moved to ws40, nodes still had spikes of IO saturation. On April 17th and 18th CPU load rose on the workspace nodes because the defence against crypto-currency miners was broken in ws40, and mining made many workspace nodes unusable. On April 20th several customers still had unreliable workspaces for workloads that had behaved normally before the April 8th release.

At times users could not start workspaces, and some lost data. On April 12th the ws-daemon rollout of a new cgroups-based IO limiter broke the CPU limiter on the ws-eu39xl cluster: all cores of each node were consumed, the services on the nodes stopped working, and the team could not reach the nodes with ssh or kubectl. Users of that XL cluster, unleashed customers and Gitpod staff, lost data. EU XL traffic was moved back to ws-eu38xl at about 13:49 UTC, but that cluster was not receiving traffic, which the team noticed at 14:21 UTC, so EU XL users could not start workspaces.

Gitpod had no way to limit IO. On April 11th it moved workspace starts back to the previous ws38 clusters, but then went forward with new clusters instead of staying there, and a breaking API change in the control plane soon made ws38 incompatible. The fixes that followed did not end the problem: the IO limiter worked in development and test but not in production, and a rewritten one still did not work as expected. A defect in the workspace networking changes of the April 8th release also made workspaces sluggish, and it was hard to find because production has IP forwarding disabled while development has it enabled.

Symptom Throughput degradationSustained high node CPUNode NotReadyData loss

Root cause

Root cause Faulty deployment

Component Node root diskContainer CPU limit
Contributing factor No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access
Trigger
The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

Detection

Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

Remediation

Action items

Lessons

Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

Sources

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Node root disk, Container CPU limit among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.