IL-0474 Harness · 2025-10-28kubernetesgcp

Harness Feature Flags auth returned 502s when rescheduled pods could not get an image deleted from Artifact Registry

One of the services that Harness Feature Flags depends on used a container image that had been deleted from Google Artifact Registry (GAR). On 28 October 2025, during a routine deployment, Kubernetes rescheduled some of that service's pods onto new nodes that did not have the image cached locally, and because the image was gone from GAR, some of the new pods could not start. The dependency was partly degraded, and from 1:47am GMT some requests to the Feature Flags authentication endpoint http://config.ff.harness.io/client/auth returned 502, which the synthetic SDK checks caught. Harness restored the image to GAR and redeployed the affected pods, and normal service resumed.

Severity: –
Scope: Some requests to the Feature Flags authentication endpoint http://config.ff.harness.io/client/auth returned 502; new SDKs starting during the incident could fail to authenticate with Harness SaaS
Users affected: SDKs starting up during the incident; SDKs that had already authenticated were not affected
Region: –
Time to detect: –
Time to resolve: –

Symptoms

At 1:47am GMT on 28 October 2025, Harness's automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing, and some requests to http://config.ff.harness.io/client/auth received 502 responses.

New SDKs that started during the incident could get 502 errors when authenticating with Harness SaaS. SDKs that had already authenticated were not affected, and up-to-date SDKs retry 5xx responses with exponential backoff, so they should have authenticated in the end. The investigation found that one of the Feature Flags service's dependencies was partly degraded because its container image was missing from Google Artifact Registry (GAR).

Symptom High error rate Image pull failures

Root cause

Root cause Data loss or corruption

Component Container registry and its credentials

Contributing factor Cache-masked failure

Trigger: During a routine deployment, Kubernetes rescheduled some pods of a Feature Flags dependency onto new nodes that did not have the image cached locally.

Detection

Automated synthetic checks that simulate SDK traffic for the Feature Flags service began failing at 1:47am GMT.

Remediation

Restored the missing image to Google Artifact Registry
Redeployed the impacted pods

Action items

Review and strengthen the image retention policies in Google Artifact Registry to prevent unintended deletions
Validate automatically that all production images are kept for a defined minimum period
Add monitoring and alerting for image pull errors

Lessons

–

Sources

https://status.harness.io/incidents/2jkj4ryrktd8 · official-status

Related cases

–

Check your cluster for this

kubegap audits a Kubernetes cluster for conditions like the ones in this case, Container registry and its credentials among them, read-only, from Claude Code or Codex. Its checks cite public reports like this one.

Open kubegap

For AI tools and scripts

This case as JSON: https://incidentlib-api.benchlet.com/cases/IL-0474
The cases most like it: https://incidentlib-api.benchlet.com/cases/IL-0474/similar
MCP server (Streamable HTTP, no key): https://incidentlib-mcp.benchlet.com/mcp

The API needs no key. The MCP server offers search_incidents, get_incident, list_incidents, get_topic and vocab; both are explained on API and MCP.