Incident Library

Organisation

Gitpod

Gitpod · gitpod

2 cases · 2022

Root cause Faulty deployment1DNS or network fault1
Component CoreDNS1Container CPU limit1Node root disk1Object storage1Container registry and its credentials1
Contributing factor Unknown limit1Limited debug access1Harmful manual action1No staged rollout1Overlapping failures1Ineffective rollback1Single point of failure1Unrepresentative staging1Gap in automated tests1
Symptom Data loss2DNS resolution failures1Image pull failures1Sustained high node CPU1Node NotReady1Service outage1Throughput degradation1

Cases

  1. IL-0591 Gitpod workspaces in all regions became IO bound and unresponsive for days after the ws39 workspace cluster releaseGitpod · 2022

    Root cause Faulty deployment

  2. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clustersGitpod · 2022

    Root cause DNS or network fault