Incident Library
Applied filters ComponentContainer registry and its credentialsFactorCache-masked failureClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry
    Harness2025
    Symptoms

    At 1:47am GMT on 28 October 2025, Harness's automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing, and some requests to http://config.ff.harness.io/client/auth received 502 responses.

    High error rateImage pull failures
    Trigger

    During a routine deployment, Kubernetes rescheduled some pods of a Feature Flags dependency onto new nodes that did not have the image cached locally.

    Root cause
    Data loss or corruption
    Component
    Container registry and its credentials
    Contributing factor
    Cache-masked failure
    Detection

    Automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing at 1:47am GMT.

    Remediation
    • Restored the missing image to Google Artifact Registry
    • Redeployed the impacted pods
    Action items
    • Review and strengthen the image retention policies in Google Artifact Registry to prevent unintended deletions
    • Validate automatically that all production images are kept for a defined minimum period
    • Add monitoring and alerting for image pull errors
    View case
  2. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel
    DigitalOcean2024
    Symptoms

    From 22:38 UTC on 28 November 2024, DigitalOcean's Cloud Control Panel stopped loading for customers, and sign-up and log-in failed. DigitalOcean declared the incident from monitoring and customer reports, an internal alert fired at 22:44 UTC, and Spaces buckets and the services that depend on them began to fail intermittently.

    DNS resolution failuresHigh error rateImage pull failures
    Trigger

    At 22:38 UTC on 28 November 2024, DigitalOcean's domain registrar, Network Solutions, made an update to the digitaloceanspaces.com domain that added an Extensible Provisioning Protocol (EPP) clientHold status code to it.

    Root cause
    Dependency failure
    Component
    Domain registrar and registryObject storageContainer registry and its credentials
    Contributing factor
    Cache-masked failureMonitoring blind spot
    Detection

    DigitalOcean declared the incident at 22:38 UTC based on monitoring and customer reports that the Cloud Control Panel would not load. An internal alert detected the issue at 22:44 UTC, and the first customer contact about the Control Panel came at 22:57 UTC.

    Remediation
    • Deployed a change that brought the Cloud Control Panel back online
    • Contacted Network Solutions to have the clientHold removed
    • Escalated to Verisign executives, the .com registry, until the clientHold was removed
    Action items
    • Work with Network Solutions to understand why the clientHold was applied to the domain
    • Review other domain registrars as possible new homes for DigitalOcean's domains
    • Review monitoring and alerting to detect registrar-imposed changes and DNS resolution problems sooner
    View case
Check your cluster for this

These 2 cases involve Container registry and its credentials and Domain registrar and registry. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause2

Data loss or corruption1 case

Component
Container registry and its credentials
Contributing factor
Cache-masked failure
Organisation
Harness
  1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry
    Harness2025

Dependency failure1 case

Component
Domain registrar and registryObject storageContainer registry and its credentials
Contributing factor
Cache-masked failureMonitoring blind spot
Organisation
DigitalOcean
  1. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel
    DigitalOcean2024