Cases5 cases

1 to 5 of 5

1. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release

   Gitpod2022

   Symptoms

   From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

   Throughput degradationSustained high node CPUNode NotReadyData loss

   Trigger

   The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

   Root cause

   Faulty deployment

   Component

   Node root diskContainer CPU limit

   Contributing factor

   No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access

   Detection

   Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

   Remediation

   • Shifted workspace starts back to the ws38 clusters on April 11th
   • Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
   • Requested a limit increase from Let's Encrypt
   • Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
   • Cleaned up workspaces stuck in a stopping state so users could start them again
   • Released ws40 with 1TB boot disks to get more IO bandwidth
   • Removed the workspaces that consumed excessive CPU time for crypto-mining
   • Blocked the fraudulent users behind the crypto-mining from using Gitpod
   • Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
   • Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
   • Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step

   Action items

   • Invest in test automation so that test results stop releases like this one
   • Vary the interval between traffic shifts to new clusters, with more time for larger releases
   • Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
   • Reduce the impact teams have on each other
   • Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
   • Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
   • Make the status page show the workspace service as down when it is, and update the playbooks
   • Keep incident communication in a single Slack channel and Google Meet
   • Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability

   Lessons

   Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

   View case
2. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters

   Gitpod2022

   Symptoms

   At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

   Image pull failuresDNS resolution failuresService outageData loss

   Root cause

   DNS or network fault

   Component

   CoreDNSContainer registry and its credentialsObject storage

   Contributing factor

   Single point of failure

   Detection

   The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

   Remediation

   • Created new workspace clusters in the EU and the US with a werft job
   • Redirected traffic to the new clusters

   Action items

   • Keep workspace data on cloud provider persistent volumes that survive node and cluster failures
   • Let users download their files from a workspace that cannot be started
   • Improve the workspace backup process
   • Use three DNS servers instead of one: a GCP-internal one, 8.8.8.8 and Cloudflare's 1.1.1.1
   • Do more DNS resiliency work, tracked in an epic
   • Improve support for multi-region clusters and later go multi-cloud

   View case
3. IL-0172 An AWS Tokyo data centre cooling failure took down 6 of 60 CockroachDB nodes on Kubernetes within 22 minutes, and ranges lost quorum because the Helm-deployed cluster had no availability zone locality configured

   Devsisters2021

   Symptoms

   During a team dinner, a Slack alert reported that one node of the CockroachDB cluster behind Cookie Run: Kingdom had gone down. The DevOps engineers were not worried at first, because CockroachDB is built to survive the loss of some nodes.

   Node NotReadyData lossService outage

   Trigger

   A cooling system failure in an AWS Tokyo region data centre overheated the servers in one availability zone, which held CockroachDB nodes.

   Root cause

   Cloud provider outage

   Component

   DatabaseHelm

   Contributing factor

   Unverified assumptionExcessive blast radius

   Detection

   A Slack alert reported that a database node had gone down.

   Remediation

   • copied the data outside the broken ranges into new tables with INSERT INTO ... SELECT queries that skipped the affected ranges
   • ran debug unsafe-remove-dead-replicas together with Cockroach Labs engineers, recovering one of the two lost data ranges
   • restored the last data range with the storage-layer recovery method from the previous incident

   Action items

   • Configure CockroachDB Locality so data is spread evenly across availability zones
   • Deploy with one chart and a separate values file per availability zone to set Locality
   • Replace nodes by adding Locality-configured nodes, confirming replicas in every zone, then retiring the old nodes

   Lessons

   A replication factor sized for random instance retirements does not protect against a whole availability zone failing; for a game of this scale a multi-AZ placement is required, not optional. Keeping one uniform Helm chart made zone topology hard to set, which is part of why it was left out. A repetitive chart with a values file per zone sets it reliably.

   View case
4. IL-0196 Adding a second Redis replica and lowering memory limits in one RedisFailover change restarted the only Redis pod that held data, and Sentinel promoted the empty new replica, erasing the data

   Palark2020

   Symptoms

   Right after an updated RedisFailover resource was rolled out, the large Redis database lost all its data and the service went down; the data had to be restored from a backup taken just before the change.

   Data lossService outage

   Trigger

   A RedisFailover change that raised the number of Redis replicas from one to two and, in the same rollout, lowered the containers' memory requests and limits from 22 GB to 19 GB.

   Root cause

   Software defect

   Component

   DatabasePersistent volumes and stateful setsContainer memory limit

   Contributing factor

   No staged rolloutSingle point of failureUnverified assumption

   Remediation

   • restored the Redis data from the backup prepared before the rollout

   Action items

   • Report the readiness probe flaw to the Redis Operator project and test the developers' fix
   • Reconsider the use of the Spotahome Redis Operator
   • Clean up the Redis databases and fix the application code that created the excess keys
   • Consider Redis built-in clustering and sharding

   Lessons

   Changing the replica count and the pod resources in one rollout turned a seemingly harmless change into a rolling restart of the only pod with data while the new replica was still empty. A readiness probe that only checks that Redis answers a ping does not tell whether a replica has finished syncing. Operators for critical stateful components need to be evaluated for maturity before production use, and the dump.rdb file should be backed up before any change to the Redis cluster.

   View case
5. IL-0506 UC Berkeley DataHub logins failed and students got empty home directories after the JupyterHub database was overwritten

   UC Berkeley2017

   Symptoms

   At about 1 AM on 9 February 2017, UC Berkeley's DataHub was reported down: users trying to log in got a proxy error. The hub pod was running, but its log was full of sqlite errors.

   Service outageData lossMisrouted requests

   Trigger

   While debugging, an operator ran `.save jupyterhub.sqlite` in sqlite3 inside the hub pod without having attached the database, which replaced the JupyterHub database file with an empty one.

   Root cause

   Operator error

   Component

   DatabasePersistent volumes and stateful setsJupyterHub

   Contributing factor

   No safeguard on destructive changeUnsafe failure behaviourHarmful manual action

   Detection

   DataHub was reported down at about 1 AM, when users got a proxy error at login. At 1:12 AM a user reported that he could log in but all his notebooks were gone.

   Remediation

   • Scaled the hub to 0 replicas
   • Rebuilt the JupyterHub sqlite database by matching Kubernetes PVC names back to user ids
   • Set cookie_id to null instead of a dummy value and attached the rebuilt database to the hub
   • Changed the cookie secret in the hub pod and restarted it to log all users out
   • Deleted all user pods to log everyone out
   • Ran a script that filled in cookie_id for all users, restarted the hub and deleted all user pods again
   • Dropped the new users with a '.' in their name and renamed the older users from '-' to '.'

   Action items

   • Report upstream that JupyterHub uses only part of the cookie secret and discards the rest
   • Report upstream that the cookie_id column of the users table should have UNIQUE and NOT NULL constraints
   • Ask KubeSpawner to support username hashes in PVC and pod names so the names stay the same when the database is deleted
   • Ask OAuthenticator to support using the id from the Google authenticator as the user id, giving stable user ids
   • Switch to Google Cloud SQL, a hosted and managed MySQL database
   • Take regular and tested backups of the database
   • Start an operational FAQ of things to do and not to do
   • Set up better monitoring and paging
   • Document escalation procedures explicitly

   View case

By root cause5