Incident Library
Applied filters Root causeSoftware defectComponentArgo CDClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. Symptoms

    After a rolling restart of the Argo CD application-controller, namespaces that several applications shared went into the Terminating state. About 97 applications were then stuck in a Missing/OutOfSync deadlock.

    Trigger

    A manual bulk sync with force: true from the Argo CD UI, run during the window after an application-controller restart, on an application whose manifests declared a namespace whose app.kubernetes.io/instance label pointed to a different application.

    Root cause
    Component
    Contributing factor
    Action items
    • Refile the problem as #29739: a manual force sync hard-replaces shared cluster-scoped resources through a tracking-ID conflict, ignoring Prune=false
    Lessons

    The sync option Prune=false only protects the prune path; it does not stop a force sync from deleting and recreating a resource. When two Argo CD applications declare the same namespace or other cluster-scoped resource, force-syncing the application that does not own the tracking label can delete it at any time.

  2. Symptoms

    This is a vulnerability report, not an outage, so nothing failed visibly. What an engineer can check is whether the Argo CD Redis server accepts connections without a password from a pod in another namespace, on port 6379 at the address kube-dns returns. In the default installation it did.

    Trigger

    Cycode Labs installed Argo CD with the default configuration on a new Amazon EKS cluster and connected to its Redis server from a low-privilege pod in another namespace, simulating a compromised pod.

    Root cause
    Component
    Contributing factor
    Detection

    Found by Cycode Labs security research, not by an operator.

    Remediation
    • Argo CD released 2.11.1, 2.10.10, 2.9.15 and 2.8.19 with an initialized password for the Redis server
    Lessons

    The hash that checks the cached manifests is computed without a secret key, so anyone who can write to Redis can recompute it and make Argo CD deploy an altered manifest. Cycode recommends upgrading to the fixed versions, checking that the argocd-redis-network-policy NetworkPolicy exists and that a CNI enforces it, using controller-based secret management instead of injecting secrets through Argo CD plugins so secrets never reach Redis, and running Argo CD on a dedicated cluster.

Check your cluster for this

These 2 cases involve Argo CD. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

By root cause1