Incident Library
Applied filters Root causeSoftware defectSymptomResources deletedClear allEvery case under the filters, with no sentence; newest first.
Suggested filters Search, and the values found in your words appear here.

Graph

  • Root cause
  • Component
  • Symptom
  • Contributing factor
  • Case

Cases2 cases

1 to 2 of 2

  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
    Symptoms

    After a rolling restart of the Argo CD application-controller, namespaces that several applications shared went into the Terminating state. About 97 applications were then stuck in a Missing/OutOfSync deadlock.

    Resources deleted
    Trigger

    A manual bulk sync with force: true from the Argo CD UI, run during the window after an application-controller restart, on an application whose manifests declared a namespace whose app.kubernetes.io/instance label pointed to a different application.

    Root cause
    Software defect
    Component
    Argo CD
    Contributing factor
    No safeguard on destructive changeUnverified assumptionHarmful manual action
    Action items
    • Refile the problem as #29739: a manual force sync hard-replaces shared cluster-scoped resources through a tracking-ID conflict, ignoring Prune=false
    Lessons

    The sync option Prune=false only protects the prune path; it does not stop a force sync from deleting and recreating a resource. When two Argo CD applications declare the same namespace or other cluster-scoped resource, force-syncing the application that does not own the tracking label can delete it at any time.

    View case
  2. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022
    Symptoms

    From 10:10 US/Pacific on 2 June 2022, Vertex AI Online Prediction returned errors for 30% up to 100% of requests per region, depending on how each customer used the service.

    High error rateResources deletedService outage
    Trigger

    A faulty resource cleanup process marked Vertex Prediction Endpoints as deleted, globally.

    Root cause
    Software defect
    Component
    Model request routing
    Contributing factor
    Automation amplificationExcessive blast radius
    Detection

    The first public status update was posted at 13:58 US/Pacific and gave the start as 10:20 US/Pacific; the later mini incident report gives 10:10.

    Remediation
    • restored the Vertex Prediction Endpoints that had been marked as deleted
    View case
Check your cluster for this

These 2 cases involve Argo CD and Model request routing. kubegap audits a Kubernetes cluster for gaps like these, read-only, from Claude Code or Codex; its checks cite public reports like the ones in this library.

Open kubegap

By root cause1

Software defect2 cases

Component
Argo CD1Model request routing1
Contributing factor
Unverified assumption1Automation amplification1Excessive blast radius1Harmful manual action1No safeguard on destructive change1
Organisation
Argo CD users1Google Cloud1
  1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync
    Argo CD users2026
  2. IL-0132 A faulty resource cleanup process marked Vertex AI prediction endpoints as deleted worldwide, undeploying customers' models and failing predictions for 4 hours 20 minutes
    Google Cloud2022