Cases3 cases

1 to 3 of 3

1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync

   Argo CD users2026

   Symptoms

   After a rolling restart of the Argo CD application-controller, namespaces that several applications shared went into the Terminating state. About 97 applications were then stuck in a Missing/OutOfSync deadlock.

   Resources deleted

   Trigger

   A manual bulk sync with force: true from the Argo CD UI, run during the window after an application-controller restart, on an application whose manifests declared a namespace whose app.kubernetes.io/instance label pointed to a different application.

   Root cause

   Software defect

   Component

   Argo CD

   Contributing factor

   No safeguard on destructive changeUnverified assumptionHarmful manual action

   Action items

   • Refile the problem as #29739: a manual force sync hard-replaces shared cluster-scoped resources through a tracking-ID conflict, ignoring Prune=false

   Lessons

   The sync option Prune=false only protects the prune path; it does not stop a force sync from deleting and recreating a resource. When two Argo CD applications declare the same namespace or other cluster-scoped resource, force-syncing the application that does not own the tracking label can delete it at any time.

   View case
2. IL-0124 Capacity changes queued while a circuit breaker was engaged ran all at once when it was released, taking multiple OpenAI inference engines out of service and raising API error rates

   OpenAI2026

   Symptoms

   Between about 4:26 PM and 4:56 PM PST on 4 March 2026, some OpenAI API requests to several models failed or took longer than usual, and some Codex traffic was affected too. The overall API success rate dropped temporarily.

   High error rateHigh latency

   Trigger

   A protective circuit breaker that had paused some scheduling operations was released, and the infrastructure actions queued during the pause were executed at the same time.

   Root cause

   Software defect

   Component

   Automated capacity manager

   Contributing factor

   Automation amplificationChange with a delayed effectNo safeguard on destructive changeMonitoring blind spot

   Remediation

   • manually reassigned infrastructure resources to the affected models
   • brought the removed inference engines back online and reassigned them until error rates returned to normal

   Action items

   • Improve safeguards in the automated capacity management systems so they detect and respond to state drift
   • Improve observability of circuit breakers and scheduling systems so their status and effects are easier to identify
   • Add safeguards so queued actions cannot run in large batches after long pauses
   • Improve monitoring and alerting around capacity reconciliation problems

   Lessons

   Pausing one automated system does not stop the systems that feed it. Changes queued during the pause reflect the state at the time they were queued, and releasing the queue all at once applies those stale decisions together. The fixes OpenAI names are detecting state drift, making circuit breaker status visible, and preventing large batches from executing after a long pause.

   View case
3. IL-0190 Cortex distributors kept a stuck connection to a dead etcd leader after a Kubernetes upgrade terminated its node, failing writes to Hosted Prometheus for 12 minutes

   Grafana Labs2020

   Symptoms

   About a minute after the etcd leader changed, two replicas of the Cortex distributor started logging "context deadline exceeded" when they tried to read keys from etcd. Writes from some customers that landed on those two replicas failed, while the other distributor replicas kept working, so the service saw an error rate of about 20% on writes to the London region of Hosted Prometheus.

   High error rateIntermittent timeouts

   Trigger

   A scheduled Kubernetes version upgrade abruptly terminated the node that ran the etcd leader at 11:19:00 UTC.

   Root cause

   Software defect

   Component

   etcdNode pool

   Contributing factor

   Unreviewed defaultsNo safeguard on destructive change

   Detection

   An SLO-based alert paged the on-call engineer at 11:31, 11 minutes after the errors started. At a 20% error rate the monthly SLA would have taken about 18 hours to breach.

   Remediation

   • Restarted the two affected Cortex distributor replicas at 11:32, which stopped the errors

   Action items

   • Enable gRPC keepalive probes in the Cortex etcd client (DialKeepAliveTime 10s, DialKeepAliveTimeout, PermitWithoutStream)
   • Make the Kubernetes upgrade process terminate pods and machines gracefully, including the etcd pods managed by the etcd operator

   Lessons

   When a node dies abruptly, clients holding TCP connections to it are not told; without keepalive probes an etcd client can keep sending requests into a dead connection long after a new leader exists. A failover that works on the server side still needs clients that detect the dead peer. Node replacement during cluster upgrades should drain pods gracefully, including pods owned by operators that the upgrade scripts do not know about, and should give instances enough time to shut down.

   View case

By root cause1