Cases1 case

1 to 1 of 1

1. IL-0437 Old pods from an unfinished Argo Rollouts canary could not start with a Helm-updated ConfigMap and died in a cascade

   Gunosy2023

   Symptoms

   During a traffic surge from a breaking news alert, pods of a Gunosy service on Kubernetes could not start because the containers could not read their settings from the ConfigMap. The pods that kept running took the extra load and were restarted, failed to start again, and pod after pod died.

   Container crash loop

   Trigger

   A breaking news alert increased traffic and pods were scaled out automatically while a release that removed an old database's settings from the ConfigMap had been rolled out to only one canary pod.

   Root cause

   Faulty deployment

   Component

   ConfigMaps and SecretsHelmProgressive delivery controller

   Contributing factor

   Partial rolloutAutomation amplification

   Action items

   • Configure Argo Rollouts to roll out automatically, with setWeight 1, a one-hour pause and then setWeight 100
   • Add the "helm.sh/resource-policy": keep annotation so Helm keeps past versions of the ConfigMap
   • Build and use a Helm plugin, helm-clear, that keeps only the last few ConfigMap versions by the revision number in their names, set to keep 5 in CD

   Lessons

   Gunosy holds that a review that only resolves to be careful next time would soon be followed by the same failure, because human attention cannot be relied on, so each incident gets a permanent fix built into the system. Keeping past ConfigMap versions also made it possible to roll back quickly without going through CI/CD, which a ConfigMap change had previously prevented.

   View case

By root cause1