Incident Library

IL-0517 Gitpod · 2022-01-25kubernetesgcpnetworking

Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters

On 25 January 2022 Gitpod had a global outage of a little over an hour. The workspace clusters used Google's primary DNS server 8.8.8.8 to resolve public addresses, and the clusters could no longer reach it: UDP traffic on port 53 timed out. Image pulls from GCR failed, so workspaces could not start, and the workspace backup service could not resolve the address of the private GCS bucket that holds workspace content backups, so backups failed and workspaces that were already running lost data. User data in /workspace was kept on the individual nodes of the workspace clusters, so a node or cluster failure before a backup finished meant data loss.

As a precaution, and because cluster creation had recently been automated, the team started creating new workspace clusters in the EU and the US in the background through a werft job, which takes about 20 minutes. The new clusters were not affected, so traffic was redirected to them and workspaces started again. The old cluster recovered shortly after traffic left it, which the team took as a sign of some kind of load issue, but later tests on the old cluster could not reproduce the network failure.

Severity
–
Scope
Workspaces could not be started, and running workspaces lost data
Users affected
–
Region
global
Time to detect
–
Time to resolve
1h

Symptoms

At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

The logs also showed "dial tcp: i/o timeout failed to do request" on image pulls, and the CoreDNS pods logged many DNS errors such as "[ERROR] plugin/errors: 2 storage.googleapis.com. A: read udp 192.168.60.206:55169->8.8.8.8:53: i/o timeout". Traffic on UDP port 53 to Google's public DNS server 8.8.8.8 seemed not to get through. Running workspaces lost data, because the backup service could not resolve the Google Cloud Storage address and their backups failed.

Google's status page showed only a Service Usage API issue that did not look related, and checks of DNS and the status page showed no errors, so the team assumed the problem was on Gitpod's side; no recent changes had been made to the clusters. The ARP table was within normal limits and conntrack utilization was at about 10%.

Symptom Image pull failuresDNS resolution failuresService outageData loss

Root cause

Root cause DNS or network fault

Component CoreDNSContainer registry and its credentialsObject storage
Contributing factor Single point of failure
Trigger
–

Detection

The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

Remediation

Action items

Lessons

–

Sources

Related cases

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, CoreDNS, Container registry and its credentials, Object storage among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.