Incident Library
Applied filters Root causeDependency failureComponentDomain registrar and registrySymptomImage pull failuresClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases1 case

1 to 1 of 1

  1. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel
    DigitalOcean2024
    Symptoms

    From 22:38 UTC on 28 November 2024, DigitalOcean's Cloud Control Panel stopped loading for customers, and sign-up and log-in failed. DigitalOcean declared the incident from monitoring and customer reports, an internal alert fired at 22:44 UTC, and Spaces buckets and the services that depend on them began to fail intermittently.

    DNS resolution failuresHigh error rateImage pull failures
    Trigger

    At 22:38 UTC on 28 November 2024, DigitalOcean's domain registrar, Network Solutions, made an update to the digitaloceanspaces.com domain that added an Extensible Provisioning Protocol (EPP) clientHold status code to it.

    Root cause
    Dependency failure
    Component
    Domain registrar and registryObject storageContainer registry and its credentials
    Contributing factor
    Cache-masked failureMonitoring blind spot
    Detection

    DigitalOcean declared the incident at 22:38 UTC based on monitoring and customer reports that the Cloud Control Panel would not load. An internal alert detected the issue at 22:44 UTC, and the first customer contact about the Control Panel came at 22:57 UTC.

    Remediation
    • Deployed a change that brought the Cloud Control Panel back online
    • Contacted Network Solutions to have the clientHold removed
    • Escalated to Verisign executives, the .com registry, until the clientHold was removed
    Action items
    • Work with Network Solutions to understand why the clientHold was applied to the domain
    • Review other domain registrars as possible new homes for DigitalOcean's domains
    • Review monitoring and alerting to detect registrar-imposed changes and DNS resolution problems sooner
    View case
Check your cluster for this

This case involves Domain registrar and registry and Object storage. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Dependency failure1 case

Component
Domain registrar and registryObject storageContainer registry and its credentials
Contributing factor
Cache-masked failureMonitoring blind spot
Organisation
DigitalOcean
  1. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel
    DigitalOcean2024