IL-0343 DigitalOcean · 2024-11-28networking

DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel

DigitalOcean's domain registrar, Network Solutions, wrongly placed an EPP clientHold on the digitaloceanspaces.com domain at 22:38 UTC on 28 November 2024. This status code stops DNS resolution for the domain until the owner contacts the registrar to clear the hold. The Cloud Control Panel stopped loading at once, and Spaces and the services built on it, including Container Registry, App Platform, Functions, Load Balancers and Mongo Managed Database backups, failed for the rest of the incident, at first intermittently and most severely from 03:00 UTC as cached DNS answers expired.

Engineers saw at 23:57 UTC that the domain's whois information had changed, and at 01:06 UTC found the clientHold in the whois record and the domain marked inactive in the Network Solutions account. DigitalOcean contacted Network Solutions at 01:08 UTC and Verisign executives, as the authoritative registry for .com domains, at 01:30 UTC. The Cloud Control Panel was brought back at 01:42 UTC. After a call and repeated escalations with Verisign executives from 02:40 UTC, the clientHold was removed at 04:03 UTC, and DigitalOcean watched its infrastructure until functionality was fully restored by 5:38:40 UTC.

Severity: –
Scope: Cloud Control Panel sign-up and log-in, Spaces objects under digitaloceanspaces.com, Spaces CDN, Container Registry create, push and pull, Mongo Managed Database backups, Load Balancer creates and resizes, App Platform builds and Functions creates and updates
Users affected: –
Region: all regions
Time to detect: 0m
Time to resolve: 7h

Symptoms

From 22:38 UTC on 28 November 2024, DigitalOcean's Cloud Control Panel stopped loading for customers, and sign-up and log-in failed. DigitalOcean declared the incident from monitoring and customer reports, an internal alert fired at 22:44 UTC, and Spaces buckets and the services that depend on them began to fail intermittently.

Every DNS record under digitaloceanspaces.com failed to resolve. Objects hosted under that domain could not be reached in any region, requests to create, push or pull container images in DigitalOcean Container Registry failed, backups for Mongo Managed Databases did not upload, and creating or resizing Load Balancers, new App Platform builds, and creating or updating Functions failed in all regions. The Cloud Control Panel depends on Spaces, which is why it went down too.

The failures were intermittent at first and grew worse over time. From 03:00 UTC on 29 November, as DNS caches and the TTLs of recursive DNS servers expired, customers saw the most severe disruption, which lasted until the domain was fixed at 04:03 UTC.

Finding the cause took hours. At 23:57 UTC engineers noticed that the whois information for the domain had been updated at 22:38 UTC, and at 01:06 UTC they found an EPP clientHold status in the whois record of digitaloceanspaces.com and saw in the Network Solutions account that the domain was marked inactive.

Symptom DNS resolution failures High error rate Image pull failures

Root cause

Root cause Dependency failure

Component Domain registrar and registry Object storage Container registry and its credentials

Contributing factor Cache-masked failure Monitoring blind spot

Trigger: At 22:38 UTC on 28 November 2024, DigitalOcean's domain registrar, Network Solutions, made an update to the digitaloceanspaces.com domain that added an Extensible Provisioning Protocol (EPP) clientHold status code to it.

Detection

DigitalOcean declared the incident at 22:38 UTC based on monitoring and customer reports that the Cloud Control Panel would not load. An internal alert detected the issue at 22:44 UTC, and the first customer contact about the Control Panel came at 22:57 UTC.

Remediation

Deployed a change that brought the Cloud Control Panel back online
Contacted Network Solutions to have the clientHold removed
Escalated to Verisign executives, the .com registry, until the clientHold was removed

Action items

Work with Network Solutions to understand why the clientHold was applied to the domain
Review other domain registrars as possible new homes for DigitalOcean's domains
Review monitoring and alerting to detect registrar-imposed changes and DNS resolution problems sooner

Lessons

–

Sources

https://status.digitalocean.com/incidents/jm44h02t22ck · official-status · archived copy

Related cases

IL-0515 Downloads from files.pythonhosted.org failed with intermittent 502 and 503 errors for two weeks, breaking PyPI installsPython Software Foundation · 2026
same root cause · shared Object storage
IL-0159 Docker Hub rate-limited Linode's IPv6 address space with 429 Too Many Requests, breaking image pulls for LKE and LKE-E clustersLinode · 2025
same root cause · shared Container registry and its credentials
IL-0126 The OPT-175B training run restarted more than 40 times in two weeks as cloud GPU nodes failed with ECC errors, replacement nodes came back faulty, and checkpoint downloads hungMeta AI · 2021
same root cause · shared Object storage

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Domain registrar and registry, Object storage, Container registry and its credentials among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0343
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0343/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.