Cases3 cases

1 to 3 of 3

1. IL-0058 Argo CD deleted namespaces shared by several applications, leaving about 97 applications stuck Missing and OutOfSync

   Argo CD users2026

   Symptoms

   After a rolling restart of the Argo CD application-controller, namespaces that several applications shared went into the Terminating state. About 97 applications were then stuck in a Missing/OutOfSync deadlock.

   Resources deleted

   Trigger

   A manual bulk sync with force: true from the Argo CD UI, run during the window after an application-controller restart, on an application whose manifests declared a namespace whose app.kubernetes.io/instance label pointed to a different application.

   Root cause

   Software defect

   Component

   Argo CD

   Contributing factor

   No safeguard on destructive changeUnverified assumptionHarmful manual action

   Action items

   • Refile the problem as #29739: a manual force sync hard-replaces shared cluster-scoped resources through a tracking-ID conflict, ignoring Prune=false

   Lessons

   The sync option Prune=false only protects the prune path; it does not stop a force sync from deleting and recreating a resource. When two Argo CD applications declare the same namespace or other cluster-scoped resource, force-syncing the application that does not own the tracking label can delete it at any time.

   View case
2. IL-0109 A retried node rollover on a DFDS EKS cluster set the Auto Scaling group back to 15 instances, and the default termination policy removed every healthy worker node, leaving the cluster with no nodes

   DFDS2023sev1

   Symptoms

   At 09:33 UTC on 14 February 2023 end users told the platform team that Kubernetes logs were not arriving in CloudWatch as expected. The investigation went through fluentd and found that the nodes were running containerd as their container runtime instead of docker-shim. A scheduled node rollover the day before, at 07:30 UTC on 13 February, had set containerd as the default runtime by accident.

   Service outageResources deletedMonitoring outage

   Trigger

   A scheduled node rollover had accidentally made containerd the default container runtime, which stopped log shipping to CloudWatch; the unscheduled rollover to revert it timed out, and the pipeline run that followed reduced the Auto Scaling group from 30 to 15 instances.

   Root cause

   Configuration change

   Component

   Container runtimeNode launch templateNode poolTerraformTelemetry agent

   Contributing factor

   Automation amplificationUnreviewed defaultsUnrepresentative stagingNo configuration testsNo runbookHarmful manual action

   Detection

   End users reported the missing CloudWatch logs that started the work. The postmortem lists 2 minutes as the time to detection of the node failure.

   Remediation

   • ran the node rollover script to add new nodes, which failed on a timeout from Docker Hub
   • ran the node rollover script a second time, which added new nodes that the load balancer marked healthy at 11:01 UTC

   Action items

   • Document the rollover process, especially what can be done when it goes wrong
   • Investigate options for improving the rollover process
   • Add fluentd log collection to the QA test suite so logs are covered end to end
   • Add alarms for fluentd when too little or no log data is sent
   • Implement a staging environment
   • Implement guards against selecting an invalid container runtime

   Lessons

   When a change goes wrong, rolling back to a known working configuration is safer than rushing a fix, which can make the damage worse. Launching new worker nodes depends on the cloud-init script not failing in any way. The QA configuration differed from production, there was no staging environment, the QA tests did not cover fluentd log collection, and the rollover process had no documentation for when it fails.

   View case
3. IL-0522 An ingress-nginx chart upgrade deleted the jenkins.io Ingress and Fastly served HTTP/503 errors for hours

   Jenkins project2021

   Symptoms

   At 11:32 UTC on 18 November 2021, a user reported in the jenkinsci/jenkins Gitter channel that https://www.jenkins.io/doc/pipeline/steps/ was answering HTTP/503 errors with the message "Guru meditation" from Fastly's varnish.

   Service outageResources deleted

   Trigger

   The ingress-nginx Helm chart was upgraded from 4.0.6 to 4.0.8 at 10:42 UTC, and the new chart added an admission webhook rule that forbids Nginx configuration blocks in Ingress annotations.

   Root cause

   Breaking change in an upgrade

   Component

   Ingress controllerAdmission webhookHelmCDN and edge network

   Contributing factor

   Unverified assumptionCache-masked failureMonitoring blind spotHarmful manual action

   Detection

   A user reported in the jenkinsci/jenkins Gitter channel at 11:32 UTC that a jenkins.io page answered HTTP/503 errors, and status.jenkins.io reported the incident at 11:49 UTC.

   Remediation

   • Disabled the automated systems, rolled the ingress-nginx chart back to 4.0.6 and deleted the admission webhook by hand
   • Removed the forbidden annotations from the jenkins.io configuration
   • Upgraded the jenkins.io Helm chart to the last version known to work with ingress-nginx 4.0.8
   • Redeployed the jenkins.io Helm chart through the automated system and told Fastly to drop all cached pages

   Action items

   • Always read the changelog in depth when updating the ingress controllers, with two people pairing in real time
   • Add custom Datadog monitoring for www.origin.jenkins.io so that such an outage is detected
   • Make the monitor send the Host header www.jenkins.io so it is not redirected to Fastly
   • Check a set of pages, such as the pipeline steps page, not only the front page
   • Consider an ephemeral k3s end-to-end test before deploying to production

   Lessons

   Trusting a patch version of a Helm chart for the Ingress controller without reading its changelog in depth let a new admission rule remove the jenkins.io Ingress in production. Without monitoring on origin.jenkins.io, a user found the outage. A complete staging cluster with the same DNS setup would have caught the problem before production.

   View case

By root cause3