Cases3 cases

1 to 3 of 3

1. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster release

   Gitpod2022

   Symptoms

   From April 11, 2022 at 13:17 UTC customers and Gitpod staff reported that Gitpod workspaces were intermittently becoming unresponsive. CPU usage of nodes and pods looked normal, with nothing maxed out, and at 13:58 UTC the team found that the workspaces were IO bound rather than CPU bound.

   Throughput degradationSustained high node CPUNode NotReadyData loss

   Trigger

   The ws39 generation of workspace clusters, released to all regional clusters on April 8th at 20:08 UTC and to the XL clusters on April 9th at 17:45 UTC, split the 1TB disk into smaller RAID0 disks, changed CPU limiting and workspace networking, and moved to faster machine types.

   Root cause

   Faulty deployment

   Component

   Node root diskContainer CPU limit

   Contributing factor

   No staged rolloutGap in automated testsUnrepresentative stagingUnknown limitOverlapping failuresIneffective rollbackHarmful manual actionLimited debug access

   Detection

   Customers and Gitpod staff reported on April 11th at 13:17 UTC that workspaces were intermittently unresponsive, and the team opened an incident in Slack through its incident.io integration. On April 17th the team noticed increased CPU load on the workspace nodes and opened an internal incident.

   Remediation

   • Shifted workspace starts back to the ws38 clusters on April 11th
   • Built ws39a clusters with c2d machines and local SSDs and moved traffic to them
   • Requested a limit increase from Let's Encrypt
   • Routed EU XL traffic to the regional EU cluster and built ws-eu39bxl to replace the crashed ws-eu39xl
   • Cleaned up workspaces stuck in a stopping state so users could start them again
   • Released ws40 with 1TB boot disks to get more IO bandwidth
   • Removed the workspaces that consumed excessive CPU time for crypto-mining
   • Blocked the fraudulent users behind the crypto-mining from using Gitpod
   • Rolled back the ws-daemon change that broke CPU limiting and shipped the fix in the ws34 clusters
   • Blocked ACME HTTP requests to workspace ports so Let's Encrypt rate limits would not be hit again
   • Rolled out IO limiting with the ws41xl and ws41 clusters and moved traffic to them step by step

   Action items

   • Invest in test automation so that test results stop releases like this one
   • Vary the interval between traffic shifts to new clusters, with more time for larger releases
   • Add a risk assessment stage, to be automated, that governs changeset size and rollout speed of workspace cluster rollouts
   • Reduce the impact teams have on each other
   • Add review criteria that find backwards incompatible changes before pull requests are merged, and split such pull requests
   • Recommend rollback as the primary means of mitigation in the incident procedures, and look at automating it
   • Make the status page show the workspace service as down when it is, and update the playbooks
   • Keep incident communication in a single Slack channel and Google Meet
   • Put 10% of the engineering staff on the on-call rotation from May 9th and use that time on observability, alerting and stability

   Lessons

   Gitpod concluded that it should have committed to rolling back on April 11th instead of building and shipping fixes, which kept customers on degraded workspaces longer than necessary. A large release so soon after the previous one needed a risk assessment and slower traffic shifts, and backwards incompatible changes that block a rollback need to be found before pull requests are merged. Failure-free operations require experience with failure.

   View case
2. IL-0328 A field type change in the database of CircleCI's job distribution service left all jobs stuck in "not running"

   CircleCI2021

   Symptoms

   From about 18:50 to 19:43 UTC on Monday, November 8, 2021, CircleCI customer jobs on all executors were blocked in a "not running" state. Errors and alerts appeared right after a deploy that changed a field type in the database of the job distribution service.

   Service outageHigh latencyScale-up failureCloud API rate limit exhausted

   Trigger

   At 18:50 UTC a deploy changed the type of a field in the PostgreSQL database of the job distribution service, in a way that was not backwards compatible.

   Root cause

   Faulty deployment

   Component

   DatabaseNode pool

   Contributing factor

   Ineffective rollbackExcessive blast radiusGap in automated tests

   Detection

   Errors and alerts related to the distribution failure appeared immediately after the 18:50 UTC deploy.

   Remediation

   • Rolled back the field type change, which did not clear the errors
   • Wrote, built and manually deployed a change that made the distribution system ignore the field
   • Manually scaled the primary compute cluster for the expected thundering herd of work
   • Monitored until processing was normal and resolved the incident

   Action items

   • Fix the gaps found in the testing strategy
   • Change how the distributor reacts to unexpected data so that this kind of deploy should not affect all jobs
   • Make recovery faster when unforeseen failures occur

   View case
3. IL-0229 bunny.net's DNS servers crashed in a loop on a corrupted routing database file and could not fetch a fix from its CDN

   bunny.net2021

   Symptoms

   On 22 June 2021, within minutes of an update released at 8:25 AM UTC, bunny.net's global fleet of close to 100 DNS servers kept crashing and restarting, more than 60% of traffic was lost, hundreds of Gbits of throughput disappeared, and over 750,000 websites were affected. After 10 minutes the team realised that the DNS servers were restarting and dying and that nothing could bring them back up in that state.

   Container crash loopDNS resolution failuresService outageMisrouted requestsLoad imbalance

   Trigger

   At 8:25 AM UTC bunny.net released an update meant to reduce the download size of the SmartEdge optimization database, and it uploaded a corrupted file to Edge Storage.

   Root cause

   Faulty deployment

   Component

   Authoritative DNS servers and recordsCDN and edge networkObject storage

   Contributing factor

   Circular dependencySingle point of failureAutomation amplificationIneffective rollbackHarmful manual action

   Detection

   About 10 minutes after the release, the team realised that the DNS servers were restarting and dying.

   Remediation

   • Deployed by hand an update that disabled the SmartEdge system on the DNS nodes
   • Routed all requests through Frankfurt instead of Madrid on a couple of DNS servers
   • Migrated the deployment systems and files to a third-party cloud storage service
   • Rewired the deployment system and DNS software to the new storage and deployed
   • Synced the databases and deployed the newest file sets through the third-party service

   Action items

   • Phase out the BinaryPack library as a hot-fix
   • Test third-party libraries more extensively
   • Migrate the internal APIs to an independent third-party service
   • Find how to stop non-critical software from becoming a single point of failure across clusters
   • Make the DNS system keep a local copy of all backup data with automatic failure detection

   Lessons

   bunny.net had built its systems to rely on each other, including the critical parts of its internal infrastructure, so one corrupted file brought down several layers of redundancy with no way back. A part of the infrastructure treated as non-critical turned out to be a single point of failure for several clusters at once, which canary deployments did not catch. Building infrastructure inside its own ecosystem lets one failure cascade through all of it. In the rush to fix the outage the team made typos and mistakes, and the final deployment came after it calmed down and double-checked everything.

   View case

By root cause1