Cases2 cases

1 to 2 of 2

1. IL-0474 Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry

   Harness2025

   Symptoms

   At 1:47am GMT on 28 October 2025, Harness's automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing, and some requests to http://config.ff.harness.io/client/auth received 502 responses.

   High error rateImage pull failures

   Trigger

   During a routine deployment, Kubernetes rescheduled some pods of a Feature Flags dependency onto new nodes that did not have the image cached locally.

   Root cause

   Data loss or corruption

   Component

   Container registry and its credentials

   Contributing factor

   Cache-masked failure

   Detection

   Automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing at 1:47am GMT.

   Remediation

   • Restored the missing image to Google Artifact Registry
   • Redeployed the impacted pods

   Action items

   • Review and strengthen the image retention policies in Google Artifact Registry to prevent unintended deletions
   • Validate automatically that all production images are kept for a defined minimum period
   • Add monitoring and alerting for image pull errors

   View case
2. IL-0343 DNS names under digitaloceanspaces.com stopped resolving, breaking Spaces, Container Registry and the Control Panel

   DigitalOcean2024

   Symptoms

   From 22:38 UTC on 28 November 2024, DigitalOcean's Cloud Control Panel stopped loading for customers, and sign-up and log-in failed. DigitalOcean declared the incident from monitoring and customer reports, an internal alert fired at 22:44 UTC, and Spaces buckets and the services that depend on them began to fail intermittently.

   DNS resolution failuresHigh error rateImage pull failures

   Trigger

   At 22:38 UTC on 28 November 2024, DigitalOcean's domain registrar, Network Solutions, made an update to the digitaloceanspaces.com domain that added an Extensible Provisioning Protocol (EPP) clientHold status code to it.

   Root cause

   Dependency failure

   Component

   Domain registrar and registryObject storageContainer registry and its credentials

   Contributing factor

   Cache-masked failureMonitoring blind spot

   Detection

   DigitalOcean declared the incident at 22:38 UTC based on monitoring and customer reports that the Cloud Control Panel would not load. An internal alert detected the issue at 22:44 UTC, and the first customer contact about the Control Panel came at 22:57 UTC.

   Remediation

   • Deployed a change that brought the Cloud Control Panel back online
   • Contacted Network Solutions to have the clientHold removed
   • Escalated to Verisign executives, the .com registry, until the clientHold was removed

   Action items

   • Work with Network Solutions to understand why the clientHold was applied to the domain
   • Review other domain registrars as possible new homes for DigitalOcean's domains
   • Review monitoring and alerting to detect registrar-imposed changes and DNS resolution problems sooner

   View case

By root cause2