Incident Library
Applied filters ComponentObject storageSymptomImage pull failuresClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel
    DigitalOcean2024
    Symptoms

    From 22:38 UTC on 28 November 2024, DigitalOcean's Cloud Control Panel stopped loading for customers, and sign-up and log-in failed. DigitalOcean declared the incident from monitoring and customer reports, an internal alert fired at 22:44 UTC, and Spaces buckets and the services that depend on them began to fail intermittently.

    DNS resolution failuresHigh error rateImage pull failures
    Trigger

    At 22:38 UTC on 28 November 2024, DigitalOcean's domain registrar, Network Solutions, made an update to the digitaloceanspaces.com domain that added an Extensible Provisioning Protocol (EPP) clientHold status code to it.

    Root cause
    Dependency failure
    Component
    Domain registrar and registryObject storageContainer registry and its credentials
    Contributing factor
    Cache-masked failureMonitoring blind spot
    Detection

    DigitalOcean declared the incident at 22:38 UTC based on monitoring and customer reports that the Cloud Control Panel would not load. An internal alert detected the issue at 22:44 UTC, and the first customer contact about the Control Panel came at 22:57 UTC.

    Remediation
    • Deployed a change that brought the Cloud Control Panel back online
    • Contacted Network Solutions to have the clientHold removed
    • Escalated to Verisign executives, the .com registry, until the clientHold was removed
    Action items
    • Work with Network Solutions to understand why the clientHold was applied to the domain
    • Review other domain registrars as possible new homes for DigitalOcean's domains
    • Review monitoring and alerting to detect registrar-imposed changes and DNS resolution problems sooner
    View case
  2. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022
    Symptoms

    At around 5.44pm UTC on 25 January 2022, Gitpod saw a high error rate when pulling images from Google Container Registry (gcr.io), with the error "cannot get manifest". About 10 minutes later users reported that they could not start workspaces.

    Image pull failuresDNS resolution failuresService outageData loss
    Root cause
    DNS or network fault
    Component
    CoreDNSContainer registry and its credentialsObject storage
    Contributing factor
    Single point of failure
    Detection

    The team noticed a high error rate on image pulls from GCR at around 5.44pm UTC; about 10 minutes later users reported that workspaces would not start, and the on-call team started incident response.

    Remediation
    • Created new workspace clusters in the EU and the US with a werft job
    • Redirected traffic to the new clusters
    Action items
    • Keep workspace data on cloud provider persistent volumes that survive node and cluster failures
    • Let users download their files from a workspace that cannot be started
    • Improve the workspace backup process
    • Use three DNS servers instead of one: a GCP-internal one, 8.8.8.8 and Cloudflare's 1.1.1.1
    • Do more DNS resiliency work, tracked in an epic
    • Improve support for multi-region clusters and later go multi-cloud
    View case
Check your cluster for this

These 2 cases involve Object storage and Container registry and its credentials. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Dependency failure1 case

Component
Domain registrar and registryObject storageContainer registry and its credentials
Contributing factor
Cache-masked failureMonitoring blind spot
Organisation
DigitalOcean
  1. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel
    DigitalOcean2024

DNS or network fault1 case

Component
CoreDNSObject storageContainer registry and its credentials
Contributing factor
Single point of failure
Organisation
Gitpod
  1. IL-0517 Gitpod workspaces failed to start worldwide and lost data when CoreDNS lookups to 8.8.8.8 timed out in its clusters
    Gitpod2022