Cases2 cases

1 to 2 of 2

1. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters

   Gitpod2022

   Symptoms

   At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

   Image pull failuresDNS resolution failuresService outageData loss

   Root cause

   DNS or network fault

   Component

   CoreDNSContainer registry and its credentialsObject storage

   Contributing factor

   Single point of failure

   Detection

   The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

   Remediation

   • Created new workspace clusters in the EU and the US with a werft job
   • Redirected traffic to the new clusters

   Action items

   • Keep workspace data on cloud provider persistent volumes that survive node and cluster failures
   • Let users download their files from a workspace that cannot be started
   • Improve the workspace backup process
   • Use three DNS servers instead of one: a GCP-internal one, 8.8.8.8 and Cloudflare's 1.1.1.1
   • Do more DNS resiliency work, tracked in an epic
   • Improve support for multi-region clusters and later go multi-cloud

   View case
2. IL-0199 A non-essential container exceeding its memory limit made GitHub's production pods unavailable, and their restarts failed because a DNS maintenance change had left the clusters unable to reach the image registry

   GitHub2020

   Symptoms

   At 08:18 UTC on 13 July 2020, production Kubernetes pods began to be marked unavailable. The loss of pods spread through the clusters and reduced capacity until GitHub.com services went down.

   OOMKilledImage pull failuresService outage

   Trigger

   A container in the production pods exceeded its memory limit and was terminated; a routine DNS maintenance operation completed earlier had left the clusters unable to reach the image registry.

   Root cause

   DNS or network fault

   Component

   Container memory limitContainer registry and its credentials

   Contributing factor

   Overlapping failuresHarmful manual actionSingle point of failure

   Remediation

   • restarted the process with the cached DNS records so the clusters could reach the registry
   • fetched the container images, redeployed and recovered the services

   Action items

   • Improve monitoring so pod restarts cannot fail again in the same pattern
   • Minimise the dependency on the image registry
   • Expand validation during DNS changes
   • Re-evaluate all existing Kubernetes deployment policies
   • Improve progressive deployments so the impact of a deployment can be evaluated incrementally

   Lessons

   A container that production traffic does not need can still take pods out of service, because a pod is available only when all of its containers are healthy. ImagePullPolicy Always turns every container restart into a registry dependency, so an unrelated DNS change that breaks registry access turns a one-minute self-healing event into a long outage, and a mass redeploy spreads it.

   View case

By root cause1